2020-03-11T22:21:26.674 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:21:26.675 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.gFackT.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848255 2020-03-11T22:21:26.695 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:21:26.754 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848255 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848255' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27429 2020-03-11T22:21:26.754 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:21:26.758 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:21:26.758 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:21:26.760 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:21:26.783 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:21:26.783 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:21:26.784 INFO:teuthology.packaging:ref: None 2020-03-11T22:21:26.784 INFO:teuthology.packaging:tag: None 2020-03-11T22:21:26.784 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:21:26.784 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:21:26.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:21:27.018 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:21:27.018 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:21:27.028 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:21:27.029 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:21:27.044 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:21:27.044 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:21:27.094 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848255', 'machine_type': 'smithi'} 2020-03-11T22:21:27.112 DEBUG:teuthology.lock.ops:locked ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:21:27.142 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:21:27.335 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of centos 8.1 2020-03-11T22:21:27.717 INFO:teuthology.orchestra.console:Power off smithi118 2020-03-11T22:21:27.718 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:21:27.765 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:27.887 INFO:teuthology.orchestra.console:Power off for smithi118 completed 2020-03-11T22:21:27.988 INFO:teuthology.orchestra.console:Power on smithi118 2020-03-11T22:21:27.988 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:21:28.018 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:32.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:36.163 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:36.375 INFO:teuthology.orchestra.console:Power on for smithi118 completed 2020-03-11T22:21:36.476 INFO:teuthology.provision.fog.smithi118:Waiting for deploy to finish 2020-03-11T22:24:22.735 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:31.802 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:40.859 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:49.911 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:58.970 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:08.031 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:17.114 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:26.170 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:35.226 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:41.230 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:41.559 INFO:teuthology.orchestra.run.smithi118:> hostname 2020-03-11T22:26:41.837 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118.front.sepia.ceph.com 2020-03-11T22:26:41.838 INFO:teuthology.orchestra.run.smithi118:> grep smithi118.front.sepia.ceph.com /etc/hosts 2020-03-11T22:26:41.854 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118 smithi118.front.sepia.ceph.com smithi118 2020-03-11T22:26:41.855 INFO:teuthology.orchestra.run.smithi118:> sudo hostname smithi118 2020-03-11T22:26:41.974 INFO:teuthology.orchestra.run.smithi118:> sudo sed -i -e 's/smithi118.front.sepia.ceph.com/smithi118/g' /etc/hosts 2020-03-11T22:26:42.008 INFO:teuthology.orchestra.run.smithi118:> sudo sed -i -e 's/172.21.15.118/172.21.15.118/g' /etc/hosts 2020-03-11T22:26:42.081 INFO:teuthology.orchestra.run.smithi118:> sudo sed -i -e 's/smithi118.front.sepia.ceph.com/smithi118/g' /etc/hostname 2020-03-11T22:26:42.152 INFO:teuthology.orchestra.run.smithi118:> sudo hostname smithi118 2020-03-11T22:26:42.222 INFO:teuthology.provision.fog.smithi118:Deploy complete! 2020-03-11T22:26:42.224 DEBUG:teuthology.parallel:result is None 2020-03-11T22:26:42.327 INFO:teuthology.lock.ops:Checking smithi118.front.sepia.ceph.com 2020-03-11T22:26:42.328 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:26:42.354 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:26:42.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:42.355 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:42.421 INFO:teuthology.orchestra.run.smithi118:> true 2020-03-11T22:26:42.588 INFO:teuthology.orchestra.run.smithi118:> uname -m 2020-03-11T22:26:42.641 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2020-03-11T22:26:42.642 INFO:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2020-03-11T22:26:42.695 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Linux" 2020-03-11T22:26:42.695 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8 (Core)" 2020-03-11T22:26:42.695 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2020-03-11T22:26:42.695 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:42.696 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:42.697 INFO:teuthology.orchestra.run.smithi118.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:42.697 INFO:teuthology.orchestra.run.smithi118.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:42.697 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:42.697 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:42.697 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:42.698 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2020-03-11T22:26:42.719 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:26:42.737 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi118.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWy571tomRfEhXi4UU/IyE6wR3z8NJX6r3NYtKMwi2pDnfFGO670s3ZzuQkT4TPumuGAU/LXWFPKAz5TNG/xiptlWnj+fxBNbyssuYBUQbJrXKDyEemYVtrNpXVa6WYFV2QxutJjBGd9gB6+gzfj7iAjxJXOAV+ff0RbCpq6+3X/jYVQJDF7lrKwISwTGzGeDPGowUoZX7vFarYqa8vAYFOwknpOpDlZZM4aW5jUvwlrTBaTCUZR2Z05Q+/ky7LSTusUdvNWMSb9yjkcdlmFbTPc9ixPPbxexSG6vDJvkRKL0aBK19yccx5yNBLTvucBnf3WtbOuSFJyqJ7/n4GD3J0RI+iqcQrBXJB4Sb+53roEDM9LKBFodq6RViVahqklspnOg2pqFoGmR/azACrmGFOCUG9ml7F8ZFJh3P1WZo/VEssraQ8pmBXcVpAizlkEnIwHAirwXHLtXw1wMvET5Iu6FlcngpCCnnYDb5Q8OOm1JpQE1j4hr6PcIzFXFb7/8= 2020-03-11T22:26:42.737 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:26:42.783 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:26:42.787 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:26:42.798 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:26:42.806 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:26:42.828 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:21:27.101986', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi118.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfMaLkeZc0qMB3wMviMvhY43iuY+GzfzOSu6b9/3Q3N/dA3RnwLqEkuWDgy7EgloIqR4qJtekluvn/dknr0OCQ=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848255'} 2020-03-11T22:26:42.829 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:26:42.840 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:26:42.840 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:26:42.880 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:26:42.881 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:26:42.890 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:26:42.890 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:26:42.891 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:42.961 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:26:42.969 INFO:teuthology.orchestra.run.smithi118:> uname -m 2020-03-11T22:26:43.120 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2020-03-11T22:26:43.121 INFO:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2020-03-11T22:26:43.174 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Linux" 2020-03-11T22:26:43.174 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8 (Core)" 2020-03-11T22:26:43.174 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2020-03-11T22:26:43.174 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:43.175 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:43.177 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2020-03-11T22:26:43.200 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:26:43.216 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:26:43.225 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:26:43.226 INFO:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:26:43.242 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:26:43.252 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:26:43.252 INFO:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:26:43.303 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:26:43.333 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:26:43.351 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:26:43.351 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:26:43.351 INFO:teuthology.orchestra.run.smithi118:> uname -r 2020-03-11T22:26:43.365 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:43.366 INFO:teuthology.task.kernel:Running kernel on smithi118: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:43.366 INFO:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2020-03-11T22:26:45.911 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 1 day, 7:00:52 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:46.305 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:26:46.306 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:26:46.456 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:26:46.457 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:26:46.457 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:26:46.597 INFO:teuthology.orchestra.run.smithi118:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:26:47.025 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 1 day, 7:00:54 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:47.406 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:26:47.407 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:47.407 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:47.407 INFO:teuthology.orchestra.run.smithi118.stdout:Reinstalling: 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:47.409 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.1 M 2020-03-11T22:26:47.409 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 0 2020-03-11T22:26:47.409 INFO:teuthology.orchestra.run.smithi118.stderr:Operation aborted. 2020-03-11T22:26:47.511 INFO:teuthology.orchestra.run.smithi118.stdout:Is this ok [y/N]: 2020-03-11T22:26:47.512 INFO:teuthology.orchestra.run.smithi118:> sudo yum reinstall -y kernel || true 2020-03-11T22:26:47.897 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 1 day, 7:00:54 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:48.254 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:26:48.255 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:48.255 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:48.255 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:48.255 INFO:teuthology.orchestra.run.smithi118.stdout:Reinstalling: 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:26:48.256 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:48.257 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.1 M 2020-03-11T22:26:48.257 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 0 2020-03-11T22:26:48.257 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:26:48.350 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): kernel-4.18.0-147.el8.x86_64.rpm 21 MB/s | 1.5 MB 00:00 2020-03-11T22:26:48.357 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 19 MB/s | 1.5 MB 00:00 2020-03-11T22:26:48.358 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:26:48.359 INFO:teuthology.orchestra.run.smithi118.stdout:Total 30 MB/s | 3.1 MB 00:00 2020-03-11T22:26:48.381 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:26:48.558 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:26:48.559 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:26:48.631 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:26:48.632 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:26:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:26:48.908 INFO:teuthology.orchestra.run.smithi118.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:26:49.034 INFO:teuthology.orchestra.run.smithi118.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:49.034 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:49.151 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:49.151 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:26:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:26:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:Reinstalled: 2020-03-11T22:26:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:26:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:26:49.729 INFO:teuthology.orchestra.run.smithi118:> rpm -q kernel --last | head -n 1 2020-03-11T22:26:49.829 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:26:48 PM UTC 2020-03-11T22:26:49.830 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:26:49.830 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:26:49.831 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:26:49.831 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:26:49.831 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:26:49.831 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:26:49.831 INFO:teuthology.orchestra.run.smithi118:> rpm -q kernel --last | head -n 1 2020-03-11T22:26:49.873 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:26:48 PM UTC 2020-03-11T22:26:49.874 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:26:49.874 INFO:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2020-03-11T22:26:49.945 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2020-03-11T22:26:49.946 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2020-03-11T22:26:49.946 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.02 2020-03-11T22:26:49.946 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 78.el8 2020-03-11T22:26:49.946 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2020-03-11T22:26:49.946 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:Group : System Environment/Base 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 9192691 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:26:49.947 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Relocations : (not relocatable) 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : CentOS BuildSystem 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2020-03-11T22:26:49.948 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2020-03-11T22:26:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:26:49.950 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:26:49.950 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:26:49.951 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:26:49.951 INFO:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:26:50.971 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2020-03-11T22:26:53.125 INFO:teuthology.orchestra.run.smithi118.stderr:done 2020-03-11T22:26:53.127 INFO:teuthology.orchestra.run.smithi118:> mktemp 2020-03-11T22:26:53.158 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.LGukMnJUru 2020-03-11T22:26:53.159 INFO:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LGukMnJUru 2020-03-11T22:26:53.241 INFO:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.LGukMnJUru 2020-03-11T22:26:53.398 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.LGukMnJUru is 4KB 2020-03-11T22:26:53.419 INFO:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.LGukMnJUru 2020-03-11T22:26:53.432 INFO:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:26:53.498 INFO:teuthology.orchestra.run.smithi118.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:26:53.499 INFO:teuthology.orchestra.run.smithi118.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:26:53.499 INFO:teuthology.orchestra.run.smithi118.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:26:53.500 INFO:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:26:53.663 INFO:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2020-03-11T22:27:23.698 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:27:23.698 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:23.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:23.698 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:42.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:42.182 DEBUG:teuthology.misc:waited 18.4841811657 2020-03-11T22:27:43.183 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:43.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:43.183 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:46.250 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:46.250 DEBUG:teuthology.misc:waited 22.5521621704 2020-03-11T22:27:47.251 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:47.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:47.251 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:49.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:49.318 DEBUG:teuthology.misc:waited 25.6201472282 2020-03-11T22:27:50.319 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:50.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:50.320 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:52.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:52.390 DEBUG:teuthology.misc:waited 28.6922271252 2020-03-11T22:27:53.391 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:53.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:53.392 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:55.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:55.462 DEBUG:teuthology.misc:waited 31.7642381191 2020-03-11T22:27:56.463 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:56.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:56.463 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:58.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:27:58.534 DEBUG:teuthology.misc:waited 34.836286068 2020-03-11T22:27:59.535 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:27:59.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:59.536 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:01.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:28:01.606 DEBUG:teuthology.misc:waited 37.9081821442 2020-03-11T22:28:02.607 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:28:02.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:02.608 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:04.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:28:04.678 DEBUG:teuthology.misc:waited 40.9802641869 2020-03-11T22:28:05.678 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:28:05.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:05.679 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:07.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:28:07.750 DEBUG:teuthology.misc:waited 44.0522310734 2020-03-11T22:28:08.751 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:28:08.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:08.752 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:10.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2020-03-11T22:28:10.822 DEBUG:teuthology.misc:waited 47.1242930889 2020-03-11T22:28:11.823 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:28:11.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:11.823 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:11.825 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:29:11.825 DEBUG:teuthology.misc:waited 108.127476215 2020-03-11T22:29:12.827 INFO:teuthology.misc:trying to connect to ubuntu@smithi118.front.sepia.ceph.com 2020-03-11T22:29:12.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:12.828 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:13.144 INFO:teuthology.orchestra.run.smithi118:> true 2020-03-11T22:29:13.436 DEBUG:teuthology.misc:waited 109.738371134 2020-03-11T22:29:14.437 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:29:14.437 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:29:14.438 INFO:teuthology.orchestra.run.smithi118:> uname -r 2020-03-11T22:29:14.455 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:29:14.456 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:29:14.456 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:29:15.457 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:29:15.457 INFO:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:29:15.549 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2020-03-11T22:29:15.561 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:29:15.568 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:29:15.568 INFO:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:29:15.584 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:29:15.699 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:29:15.769 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:29:15.770 INFO:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:29:15.795 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:29:15.844 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:29:15.845 INFO:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:29:15.881 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:29:15.892 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:29:15.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:29:15.902 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:29:15.902 INFO:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:29:15.963 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:29:15.981 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:29:15.982 INFO:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:29:16.018 INFO:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:29:16.097 INFO:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:29:16.150 INFO:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:29:16.222 INFO:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:29:16.273 INFO:teuthology.orchestra.run.smithi118:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:29:16.359 INFO:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2020-03-11T22:29:16.473 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:29:16.887 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:29:16.896 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:29:16.896 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:29:16.974 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:29:16.987 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:29:16.987 INFO:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2020-03-11T22:29:17.044 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: [ OK ] 2020-03-11T22:29:17.046 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:29:17.046 INFO:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2020-03-11T22:29:17.063 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2020-03-11T22:29:17.063 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:29:17.063 INFO:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:29:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:29:17.131 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:29:17.131 INFO:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:29:17.199 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:29:17.213 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:29:17.213 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:29:17.378 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:29:17.390 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:29:18.024 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:29:20.169 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:29:23.551 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:29:23.596 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.164) 0:00:00.164 ******* 2020-03-11T22:29:27.255 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi118.front.sepia.ceph.com] 2020-03-11T22:29:27.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:03.746) 0:00:03.910 ******* 2020-03-11T22:29:27.700 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:29:27.701 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:29:27.758 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:00.414) 0:00:04.325 ******* 2020-03-11T22:29:28.376 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:29:28.377 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:29:28.422 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:28 +0000 (0:00:00.661) 0:00:04.986 ******* 2020-03-11T22:29:28.444 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:29:28.446 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:28.482 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:28 +0000 (0:00:00.063) 0:00:05.050 ******* 2020-03-11T22:29:28.849 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:29:28.850 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:28.891 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:28 +0000 (0:00:00.408) 0:00:05.458 ******* 2020-03-11T22:29:29.919 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:29:29.920 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965768.91-236901712527774/source", "state": "file", "uid": 0} 2020-03-11T22:29:29.972 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:29 +0000 (0:00:01.080) 0:00:06.539 ******* 2020-03-11T22:29:30.478 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:29:30.479 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:29:30.480 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:29:30.577 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:29:30.582 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:29:30.785 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:00.813) 0:00:07.353 ******* 2020-03-11T22:29:31.445 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi118.front.sepia.ceph.com] 2020-03-11T22:29:31.539 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 11 March 2020 22:29:31 +0000 (0:00:00.753) 0:00:08.106 ******* 2020-03-11T22:29:32.091 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:29:32.093 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.258459", "end": "2020-03-11 22:29:32.071258", "rc": 0, "start": "2020-03-11 22:29:31.812799", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:29:32.129 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.589) 0:00:08.696 ******* 2020-03-11T22:29:32.342 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:29:32.343 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011256", "end": "2020-03-11 22:29:32.314802", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:29:32.303546", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:29:32.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.250) 0:00:08.947 ******* 2020-03-11T22:29:32.415 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:29:32.416 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:29:32.450 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.070) 0:00:09.017 ******* 2020-03-11T22:29:32.475 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:29:32.476 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.512 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.061) 0:00:09.079 ******* 2020-03-11T22:29:32.537 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:29:32.538 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.574 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.061) 0:00:09.141 ******* 2020-03-11T22:29:32.599 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:29:32.601 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.635 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.061) 0:00:09.203 ******* 2020-03-11T22:29:32.661 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:29:32.662 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.700 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.064) 0:00:09.267 ******* 2020-03-11T22:29:32.726 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:29:32.727 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.765 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.065) 0:00:09.333 ******* 2020-03-11T22:29:32.811 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:29:32.813 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.823 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.824 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.835 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.846 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.847 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.858 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.869 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.892 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.893 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.894 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.905 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.916 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.928 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.929 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.940 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.952 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.954 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.964 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.975 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.986 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.987 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.999 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.000 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.011 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.023 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.024 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.026 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.036 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.047 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.058 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.059 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.070 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.071 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.092 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.094 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.105 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.106 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.117 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.128 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.128 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.139 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.140 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.151 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.152 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.163 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.164 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.175 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.185 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.186 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.197 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.199 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.210 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.210 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.221 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.232 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.232 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.243 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.244 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.255 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.256 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.266 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.278 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.278 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.289 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.301 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.302 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.313 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.314 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.325 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.326 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.337 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.338 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.349 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.372 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.374 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.384 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.395 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.396 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.407 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.418 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.418 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.429 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.440 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.455 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.456 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.467 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.468 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.479 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.480 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.491 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.492 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.504 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.516 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.517 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.529 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.540 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.540 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.553 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.574 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.574 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.585 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.596 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.597 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.608 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.608 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.623 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.638 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.640 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.713 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.947) 0:00:10.281 ******* 2020-03-11T22:29:33.742 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:29:33.780 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.066) 0:00:10.347 ******* 2020-03-11T22:29:33.805 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:29:33.807 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.847 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.067) 0:00:10.414 ******* 2020-03-11T22:29:33.873 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:29:33.875 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.923 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.076) 0:00:10.490 ******* 2020-03-11T22:29:33.951 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:29:33.952 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.987 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.064) 0:00:10.555 ******* 2020-03-11T22:29:34.014 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.050 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.062) 0:00:10.617 ******* 2020-03-11T22:29:34.076 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.062) 0:00:10.680 ******* 2020-03-11T22:29:34.149 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:29:34.150 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.161 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.161 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.172 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.173 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.194 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.194 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.198 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.205 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.217 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.218 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.229 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.240 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.241 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.242 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.253 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.256 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.265 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.276 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.276 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.278 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.288 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.289 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.300 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.301 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.302 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.313 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.316 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.325 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.325 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.337 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.337 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.348 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.349 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.360 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.361 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.372 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.373 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.384 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.385 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.396 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:34.450 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.337) 0:00:11.017 ******* 2020-03-11T22:29:35.273 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:29:35.350 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.900) 0:00:11.917 ******* 2020-03-11T22:29:35.377 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:29:35.379 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => { "msg": "Host smithi118.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:29:35.416 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.066) 0:00:11.984 ******* 2020-03-11T22:29:35.464 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:29:35.505 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.088) 0:00:12.072 ******* 2020-03-11T22:29:35.531 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:29:35.532 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.571 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.066) 0:00:12.139 ******* 2020-03-11T22:29:35.876 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:29:35.877 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.106506", "end": "2020-03-11 22:29:35.856267", "rc": 0, "start": "2020-03-11 22:29:35.749761", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:29:35.916 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.344) 0:00:12.483 ******* 2020-03-11T22:29:35.954 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.991 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.075) 0:00:12.559 ******* 2020-03-11T22:29:36.023 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:29:36.024 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.061 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.069) 0:00:12.629 ******* 2020-03-11T22:29:36.098 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:29:36.099 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.137 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.075) 0:00:12.704 ******* 2020-03-11T22:29:36.174 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:29:36.174 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.224 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.087) 0:00:12.792 ******* 2020-03-11T22:29:36.251 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:29:36.252 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.299 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.075) 0:00:12.867 ******* 2020-03-11T22:29:36.336 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:29:36.338 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.080) 0:00:12.947 ******* 2020-03-11T22:29:36.416 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:29:36.416 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:36.418 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:36.464 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.084) 0:00:13.032 ******* 2020-03-11T22:29:36.501 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:29:36.502 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.552 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.087) 0:00:13.120 ******* 2020-03-11T22:29:36.589 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.625 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.073) 0:00:13.193 ******* 2020-03-11T22:29:36.662 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:29:36.663 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:36.723 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.098) 0:00:13.291 ******* 2020-03-11T22:29:36.760 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:29:36.762 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.804 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.080) 0:00:13.372 ******* 2020-03-11T22:29:36.840 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:29:36.842 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:36.883 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.078) 0:00:13.450 ******* 2020-03-11T22:29:36.910 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:29:36.911 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.969 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.086) 0:00:13.537 ******* 2020-03-11T22:29:37.007 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:29:37.008 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.054 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.084) 0:00:13.621 ******* 2020-03-11T22:29:37.101 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:29:37.102 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.149 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.095) 0:00:13.717 ******* 2020-03-11T22:29:37.180 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:29:37.181 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.225 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.076) 0:00:13.793 ******* 2020-03-11T22:29:37.252 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:29:37.253 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:37.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.071) 0:00:13.865 ******* 2020-03-11T22:29:37.324 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:29:37.326 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.370 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.073) 0:00:13.938 ******* 2020-03-11T22:29:37.407 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:29:37.408 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.450 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.072) 0:00:14.011 ******* 2020-03-11T22:29:37.469 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:29:37.471 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.515 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.072) 0:00:14.083 ******* 2020-03-11T22:29:37.543 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:37.545 INFO:teuthology.task.ansible.out:TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.587 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.071) 0:00:14.154 ******* 2020-03-11T22:29:37.623 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:29:37.624 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.678 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.090) 0:00:14.245 ******* 2020-03-11T22:29:37.845 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.075) 0:00:14.321 ******* TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.067) 0:00:14.389 ******* 2020-03-11T22:29:37.848 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:29:37.891 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.069) 0:00:14.458 ******* 2020-03-11T22:29:37.930 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:29:37.931 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.976 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.084) 0:00:14.543 ******* 2020-03-11T22:29:38.212 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:29:38.214 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:38.254 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.277) 0:00:14.821 ******* 2020-03-11T22:29:38.782 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:29:38.782 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:29:39.254 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:29:39.294 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:01.040) 0:00:15.862 ******* 2020-03-11T22:29:39.810 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:29:39.811 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:29:40.297 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:29:40.345 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:01.051) 0:00:16.913 ******* 2020-03-11T22:29:40.372 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:29:40.373 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.412 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.066) 0:00:16.979 ******* 2020-03-11T22:29:40.438 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:29:40.439 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.503 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.090) 0:00:17.070 ******* 2020-03-11T22:29:40.532 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:29:40.534 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.576 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.073) 0:00:17.143 ******* 2020-03-11T22:29:40.603 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:29:40.604 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.639 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.063) 0:00:17.207 ******* 2020-03-11T22:29:40.708 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:29:40.710 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.748 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.108) 0:00:17.316 ******* 2020-03-11T22:29:40.785 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:29:40.786 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.825 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.077) 0:00:17.393 ******* 2020-03-11T22:29:40.851 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:29:40.852 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.891 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.065) 0:00:17.458 ******* 2020-03-11T22:29:40.918 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:29:40.919 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.954 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.062) 0:00:17.521 ******* 2020-03-11T22:29:40.979 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:29:40.981 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:41.016 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.061) 0:00:17.583 ******* 2020-03-11T22:29:41.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:29:41.523 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.296446", "end": "2020-03-11 22:29:41.500309", "rc": 0, "start": "2020-03-11 22:29:41.203863", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:41.565 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.549) 0:00:18.132 ******* 2020-03-11T22:29:44.899 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:29:44.900 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:29:44.935 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:03.369) 0:00:21.502 ******* 2020-03-11T22:29:44.960 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:29:44.962 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:44.997 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.061) 0:00:21.564 ******* 2020-03-11T22:29:45.031 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:29:45.032 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.068 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.071) 0:00:21.636 ******* 2020-03-11T22:29:45.104 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:29:45.106 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.143 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.074) 0:00:21.710 ******* 2020-03-11T22:29:45.168 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:29:45.170 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.205 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.062) 0:00:21.772 ******* 2020-03-11T22:29:45.678 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:29:45.679 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:29:45.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.514) 0:00:22.287 ******* 2020-03-11T22:29:46.654 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:29:46.656 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:29:46.694 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.974) 0:00:23.261 ******* 2020-03-11T22:29:47.218 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:29:47.218 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:29:47.706 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:29:48.158 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:29:48.200 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:48 +0000 (0:00:01.506) 0:00:24.768 ******* 2020-03-11T22:29:48.397 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:29:48.398 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:48.441 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:48 +0000 (0:00:00.240) 0:00:25.008 ******* 2020-03-11T22:29:48.900 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:29:48.902 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:29:49.340 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:29:49.781 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:29:49.832 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:01.390) 0:00:26.399 ******* 2020-03-11T22:29:49.868 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:29:49.869 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:49.909 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.077) 0:00:26.476 ******* 2020-03-11T22:29:49.946 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:29:49.947 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:29:49.983 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.074) 0:00:26.551 ******* 2020-03-11T22:29:50.185 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:29:50.186 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:29:50.222 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.238) 0:00:26.789 ******* 2020-03-11T22:29:50.268 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:29:50.270 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.290 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.301 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.339 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.116) 0:00:26.906 ******* 2020-03-11T22:29:51.243 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:29:51.245 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:29:51.281 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:51 +0000 (0:00:00.942) 0:00:27.849 ******* 2020-03-11T22:29:51.309 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:29:51.311 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:51.346 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:51 +0000 (0:00:00.064) 0:00:27.914 ******* 2020-03-11T22:29:51.823 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:29:51.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:51 +0000 (0:00:00.520) 0:00:28.434 ******* 2020-03-11T22:29:52.393 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:29:52.394 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:29:52.438 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:00.571) 0:00:29.006 ******* 2020-03-11T22:29:52.465 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:52.501 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:00.062) 0:00:29.069 ******* 2020-03-11T22:29:52.724 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:29:52.725 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:52.760 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:00.258) 0:00:29.328 ******* 2020-03-11T22:29:53.003 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:29:53.005 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:29:53.040 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.279) 0:00:29.608 ******* 2020-03-11T22:29:53.067 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:29:53.069 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.104 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.063) 0:00:29.671 ******* 2020-03-11T22:29:53.620 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:29:53.622 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:29:53.663 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.559) 0:00:30.230 ******* 2020-03-11T22:29:54.450 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:29:54.452 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:28:41 UTC", "ActiveEnterTimestampMonotonic": "11516956", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service system.slice sysinit.target network.target -.mount systemd-journald.socket nss-lookup.target basic.target var-run.mount time-sync.target tmp.mount local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:28:41 UTC", "AssertTimestampMonotonic": "11515906", "Before": "xdm.service plymouth-quit.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:28:41 UTC", "ConditionTimestampMonotonic": "11515906", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:29:54.453 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2403", "ExecMainStartTimestamp": "Wed 2020-03-11 22:28:41 UTC", "ExecMainStartTimestampMonotonic": "11516919", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:28:41 UTC] ; stop_time=[n/a] ; pid=2403 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:28:41 UTC", "InactiveExitTimestampMonotonic": "11516956", "InvocationID": "2f3729268d944a44aa89e26aeb45242a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:29:54.454 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2403", "MemoryAccounting": "yes", "MemoryCurrent": "1916928", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:29:54.456 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:28:41 UTC", "StateChangeTimestampMonotonic": "11516956", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:28:41 UTC", "WatchdogTimestampMonotonic": "11516955", "WatchdogUSec": "0"}} 2020-03-11T22:29:54.492 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.829) 0:00:31.060 ******* 2020-03-11T22:29:54.701 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:29:54.701 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001739", "end": "2020-03-11 22:29:54.671840", "rc": 0, "start": "2020-03-11 22:29:54.670101", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:29:54.742 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.250) 0:00:31.310 ******* 2020-03-11T22:29:55.659 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:29:55.660 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:29:55.716 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:55 +0000 (0:00:00.973) 0:00:32.283 ******* 2020-03-11T22:29:56.658 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:29:56.659 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:29:56.702 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:56 +0000 (0:00:00.986) 0:00:33.270 ******* 2020-03-11T22:29:58.309 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-11T22:29:58.310 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:29:58.362 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:58 +0000 (0:00:01.658) 0:00:34.929 ******* 2020-03-11T22:30:14.627 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:30:14.629 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.070879", "end": "2020-03-11 22:30:14.603919", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:29:58.533040", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:30:14.665 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:14 +0000 (0:00:16.303) 0:00:51.232 ******* 2020-03-11T22:30:15.145 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:30:15.146 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:30:15.188 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:15 +0000 (0:00:00.523) 0:00:51.756 ******* 2020-03-11T22:30:15.415 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:30:15.416 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.029380", "end": "2020-03-11 22:30:15.390667", "rc": 0, "start": "2020-03-11 22:30:15.361287", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:15.454 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:15 +0000 (0:00:00.266) 0:00:52.022 ******* 2020-03-11T22:30:15.653 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:30:15.653 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002135", "end": "2020-03-11 22:30:15.627209", "rc": 0, "start": "2020-03-11 22:30:15.625074", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:15.692 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:15 +0000 (0:00:00.237) 0:00:52.260 ******* 2020-03-11T22:30:31.843 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:30:31.845 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.959588", "end": "2020-03-11 22:30:31.821513", "rc": 0, "start": "2020-03-11 22:30:15.861925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:31.885 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:31 +0000 (0:00:16.193) 0:01:08.453 ******* 2020-03-11T22:30:32.086 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:30:32.087 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:30:32.126 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.240) 0:01:08.693 ******* 2020-03-11T22:30:32.162 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:30:32.163 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:30:32.208 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.078) 0:01:08.771 ******* 2020-03-11T22:30:32.253 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:30:32.254 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:30:32.297 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.092) 0:01:08.864 ******* 2020-03-11T22:30:32.355 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:30:32.357 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:30:32.403 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.106) 0:01:08.971 ******* 2020-03-11T22:30:32.441 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:30:32.443 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:30:32.487 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.083) 0:01:09.054 ******* 2020-03-11T22:30:32.710 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:30:32.712 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:30:32.754 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.267) 0:01:09.321 ******* 2020-03-11T22:30:32.956 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:30:32.957 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:30:32.997 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:00.243) 0:01:09.565 ******* 2020-03-11T22:30:33.023 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:30:33.025 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:33.081 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:33 +0000 (0:00:00.083) 0:01:09.649 ******* 2020-03-11T22:30:33.109 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:30:33.111 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:33.156 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:33 +0000 (0:00:00.075) 0:01:09.724 ******* 2020-03-11T22:30:33.377 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:30:33.378 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:30:33.417 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:33 +0000 (0:00:00.261) 0:01:09.985 ******* 2020-03-11T22:30:34.098 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:30:34.099 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:30:34.137 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:34 +0000 (0:00:00.719) 0:01:10.704 ******* 2020-03-11T22:30:34.173 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:30:34.178 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi118"}, "changed": false} 2020-03-11T22:30:34.210 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:34 +0000 (0:00:00.073) 0:01:10.778 ******* 2020-03-11T22:30:35.217 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:30:35.219 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi118.front.sepia.ceph.com", "ansible_hostname": "smithi118", "ansible_nodename": "smithi118"}, "changed": true, "name": "smithi118"} 2020-03-11T22:30:35.258 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:35 +0000 (0:00:01.047) 0:01:11.825 ******* 2020-03-11T22:30:35.718 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:30:35.719 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:30:35.762 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:35 +0000 (0:00:00.504) 0:01:12.330 ******* 2020-03-11T22:30:36.244 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:30:36.244 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:30:36.283 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:36 +0000 (0:00:00.520) 0:01:12.850 ******* 2020-03-11T22:30:36.866 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:30:36.868 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:30:36.902 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:36 +0000 (0:00:00.619) 0:01:13.470 ******* 2020-03-11T22:30:37.400 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:30:37.402 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:30:37.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:37 +0000 (0:00:00.536) 0:01:14.006 ******* 2020-03-11T22:30:37.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:30:37.641 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:30:37.677 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:37 +0000 (0:00:00.238) 0:01:14.245 ******* 2020-03-11T22:30:37.880 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:30:37.882 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:30:37.920 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:37 +0000 (0:00:00.242) 0:01:14.487 ******* 2020-03-11T22:30:38.142 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:30:38.144 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:30:38.181 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:38 +0000 (0:00:00.260) 0:01:14.749 ******* 2020-03-11T22:30:38.653 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:30:38.654 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:30:38.694 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:38 +0000 (0:00:00.513) 0:01:15.262 ******* 2020-03-11T22:30:39.228 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:30:39.229 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:30:39.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:39 +0000 (0:00:00.575) 0:01:15.838 ******* 2020-03-11T22:30:41.153 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:30:41.153 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:30:41.154 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.686093", "end": "2020-03-11 22:30:41.126984", "rc": 0, "start": "2020-03-11 22:30:39.440891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:41.196 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:41 +0000 (0:00:01.925) 0:01:17.764 ******* 2020-03-11T22:30:41.717 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:30:41.718 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:30:42.174 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:30:42.639 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:30:42.685 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:42 +0000 (0:00:01.488) 0:01:19.252 ******* 2020-03-11T22:30:43.196 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:30:43.196 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:30:43.242 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:43 +0000 (0:00:00.557) 0:01:19.810 ******* 2020-03-11T22:30:43.745 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:30:43.745 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:30:44.188 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:30:44.643 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:30:44.685 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:44 +0000 (0:00:01.442) 0:01:21.252 ******* 2020-03-11T22:30:45.242 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:30:45.243 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.343675", "end": "2020-03-11 22:30:45.214191", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:30:44.870516", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:30:45.245 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:30:45.280 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.595) 0:01:21.848 ******* 2020-03-11T22:30:45.306 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:30:45.308 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.062) 0:01:21.911 ******* 2020-03-11T22:30:45.380 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:30:45.381 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.392 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.085) 0:01:21.996 ******* 2020-03-11T22:30:45.455 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:30:45.457 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.493 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.063) 0:01:22.060 ******* 2020-03-11T22:30:45.530 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:30:45.532 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:30:45.568 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.075) 0:01:22.136 ******* 2020-03-11T22:30:45.604 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:30:45.606 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.643 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.074) 0:01:22.210 ******* 2020-03-11T22:30:45.679 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:30:45.681 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.717 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.074) 0:01:22.285 ******* 2020-03-11T22:30:45.929 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:30:45.930 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006389", "end": "2020-03-11 22:30:45.902494", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:30:45.896105", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:30:45.970 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.252) 0:01:22.538 ******* 2020-03-11T22:30:46.006 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:30:46.007 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:46.045 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:46 +0000 (0:00:00.075) 0:01:22.613 ******* 2020-03-11T22:30:48.471 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:30:48.472 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:30:48.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:02.496) 0:01:25.109 ******* 2020-03-11T22:30:51.569 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:30:51.571 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:30:51.606 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:03.064) 0:01:28.174 ******* 2020-03-11T22:30:56.997 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:30:56.999 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:05.169166", "end": "2020-03-11 22:30:56.972216", "rc": 0, "start": "2020-03-11 22:30:51.803050", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 139 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 116 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 46 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 \nlab-extras 1.1 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:30:52 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:30:57.000 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 139 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 116 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 46 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 ", "lab-extras 1.1 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:30:52 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:30:57.041 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:57 +0000 (0:00:05.434) 0:01:33.609 ******* 2020-03-11T22:31:04.670 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:31:04.671 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:31:04.709 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:04 +0000 (0:00:07.667) 0:01:41.276 ******* 2020-03-11T22:31:04.745 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:31:04.746 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:04.799 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:04 +0000 (0:00:00.089) 0:01:41.366 ******* 2020-03-11T22:31:05.764 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:31:05.765 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:31:05.800 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:01.001) 0:01:42.368 ******* 2020-03-11T22:31:05.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:31:05.838 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:05.896 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.096) 0:01:42.464 ******* 2020-03-11T22:31:06.220 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:31:06.222 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service sysinit.target system.slice dbus.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service ip6tables.service ipset.service shutdown.target ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:31:06.224 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", 2020-03-11T22:31:06.225 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:31:06.227 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:31:06.265 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.368) 0:01:42.833 ******* 2020-03-11T22:31:06.560 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:31:06.561 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:31:06.597 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.331) 0:01:43.164 ******* 2020-03-11T22:31:07.398 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:31:07.399 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:31:07.436 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.839) 0:01:44.003 ******* 2020-03-11T22:31:07.461 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:31:07.463 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.498 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.062) 0:01:44.065 ******* 2020-03-11T22:31:07.523 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:31:07.560 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.062) 0:01:44.127 ******* 2020-03-11T22:31:07.585 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:31:07.587 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.622 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.062) 0:01:44.190 ******* 2020-03-11T22:31:07.648 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:31:07.649 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.686 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.063) 0:01:44.254 ******* 2020-03-11T22:31:07.722 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:31:07.724 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.762 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.075) 0:01:44.329 ******* 2020-03-11T22:31:07.787 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:31:07.788 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.826 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.063) 0:01:44.393 ******* 2020-03-11T22:31:07.851 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:31:07.890 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.064) 0:01:44.458 ******* 2020-03-11T22:31:07.926 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:31:07.927 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:07.963 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.072) 0:01:44.530 ******* 2020-03-11T22:31:07.990 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:31:07.991 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.027 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.063) 0:01:44.594 ******* 2020-03-11T22:31:08.053 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:31:08.054 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.091 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.064) 0:01:44.659 ******* 2020-03-11T22:31:08.117 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:31:08.119 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.155 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.063) 0:01:44.722 ******* 2020-03-11T22:31:08.181 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:31:08.182 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.217 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.061) 0:01:44.784 ******* 2020-03-11T22:31:08.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:31:08.243 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.280 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:44.847 ******* 2020-03-11T22:31:08.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:31:08.307 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.063) 0:01:44.910 ******* 2020-03-11T22:31:08.368 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:31:08.370 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.406 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:44.973 ******* 2020-03-11T22:31:08.431 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:31:08.433 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.467 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.061) 0:01:45.035 ******* 2020-03-11T22:31:08.494 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:31:08.495 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.531 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.064) 0:01:45.099 ******* 2020-03-11T22:31:08.557 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:31:08.558 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.594 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:45.162 ******* 2020-03-11T22:31:08.620 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:31:08.621 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.657 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:45.224 ******* 2020-03-11T22:31:08.682 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:31:08.684 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:45.287 ******* 2020-03-11T22:31:08.745 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:31:08.747 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.787 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.067) 0:01:45.355 ******* 2020-03-11T22:31:08.813 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:31:08.814 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.852 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.064) 0:01:45.419 ******* 2020-03-11T22:31:08.877 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:31:08.879 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.920 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.068) 0:01:45.488 ******* 2020-03-11T22:31:08.946 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:31:08.947 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:08.983 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.062) 0:01:45.550 ******* 2020-03-11T22:31:09.009 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:31:09.010 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.049 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.066) 0:01:45.617 ******* 2020-03-11T22:31:09.074 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:31:09.076 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.116 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.067) 0:01:45.684 ******* 2020-03-11T22:31:09.143 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:31:09.145 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.182 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.064) 0:01:45.749 ******* 2020-03-11T22:31:09.207 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:31:09.209 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.273 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.091) 0:01:45.841 ******* 2020-03-11T22:31:09.300 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:31:09.302 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.338 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.064) 0:01:45.905 ******* 2020-03-11T22:31:09.363 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:31:09.365 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.415 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.076) 0:01:45.982 ******* 2020-03-11T22:31:09.441 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:31:09.442 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.483 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.067) 0:01:46.050 ******* 2020-03-11T22:31:09.509 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:31:09.510 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.546 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.062) 0:01:46.113 ******* 2020-03-11T22:31:09.572 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.609 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.063) 0:01:46.177 ******* 2020-03-11T22:31:09.831 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:31:09.832 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:31:09.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.261) 0:01:46.438 ******* 2020-03-11T22:31:10.094 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:31:10.096 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:31:10.135 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.264) 0:01:46.702 ******* 2020-03-11T22:31:10.347 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:31:10.348 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:31:10.386 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.251) 0:01:46.954 ******* 2020-03-11T22:31:10.412 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:31:10.414 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.449 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.062) 0:01:47.017 ******* 2020-03-11T22:31:10.475 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:31:10.476 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.512 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.062) 0:01:47.079 ******* 2020-03-11T22:31:10.538 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:31:10.540 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.066) 0:01:47.145 ******* 2020-03-11T22:31:10.603 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:31:10.605 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.639 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.061) 0:01:47.207 ******* 2020-03-11T22:31:10.665 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:31:10.666 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.711 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.071) 0:01:47.279 ******* 2020-03-11T22:31:10.749 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:31:10.750 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.787 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.075) 0:01:47.354 ******* 2020-03-11T22:31:10.812 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:31:10.814 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.848 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.061) 0:01:47.416 ******* 2020-03-11T22:31:10.874 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:31:10.876 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.910 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.061) 0:01:47.477 ******* 2020-03-11T22:31:10.958 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:31:10.959 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.003 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.092) 0:01:47.570 ******* 2020-03-11T22:31:11.028 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:31:11.030 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.066 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.062) 0:01:47.633 ******* 2020-03-11T22:31:11.215 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:31:11.217 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.366 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.300) 0:01:47.933 ******* 2020-03-11T22:31:11.392 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:31:11.393 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.063) 0:01:47.997 ******* 2020-03-11T22:31:11.455 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:31:11.457 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.494 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.064) 0:01:48.062 ******* 2020-03-11T22:31:11.530 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:31:11.531 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.579 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.084) 0:01:48.146 ******* 2020-03-11T22:31:11.607 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:31:11.608 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.651 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.072) 0:01:48.219 ******* 2020-03-11T22:31:11.677 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:31:11.679 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.716 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.064) 0:01:48.284 ******* 2020-03-11T22:31:11.745 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.783 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.067) 0:01:48.351 ******* 2020-03-11T22:31:11.809 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:31:11.810 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.846 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.062) 0:01:48.414 ******* 2020-03-11T22:31:11.872 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:31:11.873 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.912 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.065) 0:01:48.479 ******* 2020-03-11T22:31:11.938 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:31:11.939 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.063) 0:01:48.543 ******* 2020-03-11T22:31:12.004 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:31:12.005 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.042 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.066) 0:01:48.609 ******* 2020-03-11T22:31:12.068 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:31:12.069 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.108 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.066) 0:01:48.676 ******* 2020-03-11T22:31:12.135 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:12.137 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:31:12.174 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.065) 0:01:48.742 ******* 2020-03-11T22:31:12.220 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:31:12.222 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.242 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:31:12.253 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.264 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.275 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.275 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.286 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.327 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.152) 0:01:48.895 ******* 2020-03-11T22:31:12.590 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi118.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.013128", "end": "2020-03-11 22:31:12.561138", "rc": 0, "start": "2020-03-11 22:31:12.548010", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:31:12.637 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.310) 0:01:49.205 ******* 2020-03-11T22:31:12.673 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:31:12.675 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.724 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.087) 0:01:49.292 ******* 2020-03-11T22:31:12.762 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:31:12.764 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.809 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.084) 0:01:49.376 ******* 2020-03-11T22:31:12.837 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:31:12.838 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.879 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.070) 0:01:49.446 ******* 2020-03-11T22:31:12.905 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:31:12.906 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.953 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.074) 0:01:49.520 ******* 2020-03-11T22:31:13.880 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:31:13.882 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:31:13.918 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.964) 0:01:50.485 ******* 2020-03-11T22:31:13.954 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:31:13.955 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:31:13.992 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.074) 0:01:50.560 ******* 2020-03-11T22:31:14.192 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:31:14.194 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005135", "end": "2020-03-11 22:31:14.166243", "rc": 0, "start": "2020-03-11 22:31:14.161108", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:14.239 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.246) 0:01:50.806 ******* 2020-03-11T22:31:14.285 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:31:14.286 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.325 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.085) 0:01:50.892 ******* 2020-03-11T22:31:14.350 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:31:14.386 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.061) 0:01:50.954 ******* 2020-03-11T22:31:14.445 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:31:14.446 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QARN', u'wwn-0x5000c50091e0b046'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QARN', u'wwn': u'0x5000c50091e0b046', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QARN-part1', u'wwn-0x5000c50091e0b046-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN", "wwn-0x5000c50091e0b046"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN-part1", "wwn-0x5000c50091e0b046-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QARN", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:31:14.447 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50091e0b046"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.017 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Bck1lu-EzIh-1dyZ-P8TO-c8qC-XXFD-wQ572B', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204011J400BGN', u'nvme-nvme.8086-50484654363230343031314a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.382441", "end": "2020-03-11 22:31:15.984754", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Bck1lu-EzIh-1dyZ-P8TO-c8qC-XXFD-wQ572B", "nvme-INTEL_SSDPEDMD400G4_PHFT6204011J400BGN", "nvme-nvme.8086-50484654363230343031314a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:14.602313", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:31:16.019 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:17.268 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSVCo0Tp6o1VSgJXujZ2K8Nn6tGmuDA1'], u'uuids': [u'0fb29220-3def-4d25-a14c-597e3cd56c02']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.055035", "end": "2020-03-11 22:31:17.235683", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSVCo0Tp6o1VSgJXujZ2K8Nn6tGmuDA1"], "labels": [], "masters": [], "uuids": ["0fb29220-3def-4d25-a14c-597e3cd56c02"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:16.180648", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:18.517 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1YPUiXw1sxKHxKfHDc24ikm8S78t4k4o9'], u'uuids': [u'a73bafaf-7ffa-4f5d-a4cf-c3ec6831ec0c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.057853", "end": "2020-03-11 22:31:18.485558", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1YPUiXw1sxKHxKfHDc24ikm8S78t4k4o9"], "labels": [], "masters": [], "uuids": ["a73bafaf-7ffa-4f5d-a4cf-c3ec6831ec0c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:17.427705", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:19.768 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSbI8fhdHcmswlDZCPX0xW0lZerdxsNV'], u'uuids': [u'84978635-3aa6-4c1b-8fa0-3078cbe70c59']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.050078", "end": "2020-03-11 22:31:19.739642", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSbI8fhdHcmswlDZCPX0xW0lZerdxsNV"], "labels": [], "masters": [], "uuids": ["84978635-3aa6-4c1b-8fa0-3078cbe70c59"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:18.689564", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:21.024 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12DhFWh43kxFGsZgXxSaZ6QACEWtjqQGA'], u'uuids': [u'f3bf10d1-8c1a-42e9-9149-8f860d966e9f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.047112", "end": "2020-03-11 22:31:20.985241", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12DhFWh43kxFGsZgXxSaZ6QACEWtjqQGA"], "labels": [], "masters": [], "uuids": ["f3bf10d1-8c1a-42e9-9149-8f860d966e9f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:19.938129", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:22.286 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12YOleoKKZRmrwx0dBXATjdDgEasGtuSL'], u'uuids': [u'f08cb25f-74b8-4645-b599-e111bb9b8a38']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.047411", "end": "2020-03-11 22:31:22.244416", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12YOleoKKZRmrwx0dBXATjdDgEasGtuSL"], "labels": [], "masters": [], "uuids": ["f08cb25f-74b8-4645-b599-e111bb9b8a38"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:21.197005", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:22.329 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:22 +0000 (0:00:07.942) 0:01:58.896 ******* 2020-03-11T22:31:22.367 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:31:22.368 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QARN', u'wwn-0x5000c50091e0b046'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QARN', u'wwn': u'0x5000c50091e0b046', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QARN-part1', u'wwn-0x5000c50091e0b046-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN", "wwn-0x5000c50091e0b046"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN-part1", "wwn-0x5000c50091e0b046-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QARN", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:31:22.369 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50091e0b046"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:22.598 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Bck1lu-EzIh-1dyZ-P8TO-c8qC-XXFD-wQ572B', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204011J400BGN', u'nvme-nvme.8086-50484654363230343031314a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037153", "end": "2020-03-11 22:31:22.570220", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Bck1lu-EzIh-1dyZ-P8TO-c8qC-XXFD-wQ572B", "nvme-INTEL_SSDPEDMD400G4_PHFT6204011J400BGN", "nvme-nvme.8086-50484654363230343031314a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:22.533067", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-11T22:31:22.600 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0345697 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0345697 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:22.940 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSVCo0Tp6o1VSgJXujZ2K8Nn6tGmuDA1'], u'uuids': [u'0fb29220-3def-4d25-a14c-597e3cd56c02']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145033", "end": "2020-03-11 22:31:22.909837", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSVCo0Tp6o1VSgJXujZ2K8Nn6tGmuDA1"], "labels": [], "masters": [], "uuids": ["0fb29220-3def-4d25-a14c-597e3cd56c02"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:22.764804", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.1425 s, 809 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.1425 s, 809 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:23.271 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1YPUiXw1sxKHxKfHDc24ikm8S78t4k4o9'], u'uuids': [u'a73bafaf-7ffa-4f5d-a4cf-c3ec6831ec0c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.147651", "end": "2020-03-11 22:31:23.243547", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1YPUiXw1sxKHxKfHDc24ikm8S78t4k4o9"], "labels": [], "masters": [], "uuids": ["a73bafaf-7ffa-4f5d-a4cf-c3ec6831ec0c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:23.095896", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145033 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145033 s, 795 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:23.605 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSbI8fhdHcmswlDZCPX0xW0lZerdxsNV'], u'uuids': [u'84978635-3aa6-4c1b-8fa0-3078cbe70c59']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.138172", "end": "2020-03-11 22:31:23.572534", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl1LSbI8fhdHcmswlDZCPX0xW0lZerdxsNV"], "labels": [], "masters": [], "uuids": ["84978635-3aa6-4c1b-8fa0-3078cbe70c59"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:23.434362", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135698 s, 850 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135698 s, 850 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:23.927 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12DhFWh43kxFGsZgXxSaZ6QACEWtjqQGA'], u'uuids': [u'f3bf10d1-8c1a-42e9-9149-8f860d966e9f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140728", "end": "2020-03-11 22:31:23.894055", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12DhFWh43kxFGsZgXxSaZ6QACEWtjqQGA"], "labels": [], "masters": [], "uuids": ["f3bf10d1-8c1a-42e9-9149-8f860d966e9f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:23.753327", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138155 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138155 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:24.254 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12YOleoKKZRmrwx0dBXATjdDgEasGtuSL'], u'uuids': [u'f08cb25f-74b8-4645-b599-e111bb9b8a38']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139818", "end": "2020-03-11 22:31:24.218445", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-4v0Nhz9UE6jUhSfp8qSeaQpjvVptTQl12YOleoKKZRmrwx0dBXATjdDgEasGtuSL"], "labels": [], "masters": [], "uuids": ["f08cb25f-74b8-4645-b599-e111bb9b8a38"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:24.078627", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137238 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137238 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:24.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:24 +0000 (0:00:01.969) 0:02:00.866 ******* 2020-03-11T22:31:24.553 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:31:24.554 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.053986", "end": "2020-03-11 22:31:24.524482", "rc": 0, "start": "2020-03-11 22:31:24.470496", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:24.591 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:24 +0000 (0:00:00.292) 0:02:01.158 ******* 2020-03-11T22:31:24.844 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:31:24.845 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048358", "end": "2020-03-11 22:31:24.816643", "rc": 0, "start": "2020-03-11 22:31:24.768285", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:24.890 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:24 +0000 (0:00:00.299) 0:02:01.458 ******* 2020-03-11T22:31:24.919 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:31:24.959 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:24 +0000 (0:00:00.069) 0:02:01.527 ******* 2020-03-11T22:31:24.985 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:31:24.986 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.026 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.066) 0:02:01.593 ******* 2020-03-11T22:31:25.052 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:31:25.053 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.097 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.070) 0:02:01.663 ******* 2020-03-11T22:31:25.134 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:31:25.135 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.205 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.108) 0:02:01.772 ******* 2020-03-11T22:31:25.231 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:31:25.232 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.279 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.074) 0:02:01.846 ******* 2020-03-11T22:31:25.507 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:31:25.508 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:31:25.558 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.278) 0:02:02.125 ******* 2020-03-11T22:31:25.609 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:31:25.610 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.666 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.107) 0:02:02.233 ******* 2020-03-11T22:31:25.704 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.742 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.076) 0:02:02.309 ******* 2020-03-11T22:31:25.769 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:25.770 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.814 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.071) 0:02:02.381 ******* 2020-03-11T22:31:25.851 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:25.853 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.900 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.086) 0:02:02.468 ******* 2020-03-11T22:31:25.927 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:31:25.929 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:25.981 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.080) 0:02:02.548 ******* 2020-03-11T22:31:26.018 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:31:26.019 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:26.062 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:26 +0000 (0:00:00.081) 0:02:02.630 ******* 2020-03-11T22:31:26.089 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:31:26.090 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:26.126 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:26 +0000 (0:00:00.063) 0:02:02.694 ******* 2020-03-11T22:31:26.763 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:31:26.764 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:31:26.803 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:26 +0000 (0:00:00.676) 0:02:03.371 ******* 2020-03-11T22:31:27.339 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:31:27.340 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:27.735 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:28.066 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:28.398 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:28.740 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:28.773 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:28 +0000 (0:00:01.969) 0:02:05.340 ******* 2020-03-11T22:31:28.985 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:31:28.987 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:31:29.029 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.256) 0:02:05.597 ******* 2020-03-11T22:31:29.080 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:31:29.080 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:29.279 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:29.467 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:29.657 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:29.847 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:29.891 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.862) 0:02:06.459 ******* 2020-03-11T22:31:30.104 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:31:30.106 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:31:30.141 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.250) 0:02:06.709 ******* 2020-03-11T22:31:30.181 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:31:30.243 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.101) 0:02:06.810 ******* 2020-03-11T22:31:30.765 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:31:30.766 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:31:30.811 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.568) 0:02:07.378 ******* 2020-03-11T22:31:31.305 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:31:31.306 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:31:31.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.532) 0:02:07.911 ******* 2020-03-11T22:31:31.370 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:31:31.372 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.409 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.064) 0:02:07.975 ******* 2020-03-11T22:31:31.435 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:31:31.437 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.473 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.065) 0:02:08.041 ******* 2020-03-11T22:31:31.499 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.536 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.062) 0:02:08.103 ******* 2020-03-11T22:31:31.562 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:31:31.563 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.603 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.066) 0:02:08.170 ******* 2020-03-11T22:31:31.629 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:31:31.630 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.674 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.071) 0:02:08.241 ******* 2020-03-11T22:31:31.700 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:31:31.701 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.738 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.063) 0:02:08.305 ******* 2020-03-11T22:31:31.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:31:31.765 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.800 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.062) 0:02:08.368 ******* 2020-03-11T22:31:31.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:31:31.838 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.074) 0:02:08.442 ******* 2020-03-11T22:31:31.900 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:31:31.902 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:31.937 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.062) 0:02:08.504 ******* 2020-03-11T22:31:31.963 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:31:31.964 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:32.001 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.063) 0:02:08.568 ******* 2020-03-11T22:31:32.027 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:31:32.028 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:32.064 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:32 +0000 (0:00:00.063) 0:02:08.632 ******* 2020-03-11T22:31:32.553 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:31:32.555 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:31:32.592 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:32 +0000 (0:00:00.527) 0:02:09.159 ******* 2020-03-11T22:31:32.898 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:31:32.899 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:28:41 UTC", "ActiveEnterTimestampMonotonic": "11367120", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service tmp.mount sntp.service sysinit.target -.mount ntpdate.service systemd-journald.socket ntpd.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:28:39 UTC", "AssertTimestampMonotonic": "10209135", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:28:39 UTC", "ConditionTimestampMonotonic": "10209112", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:31:32.901 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2335", "ExecMainStartTimestamp": "Wed 2020-03-11 22:28:39 UTC", "ExecMainStartTimestampMonotonic": "10316832", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:28:39 UTC", "InactiveExitTimestampMonotonic": "10210806", "InvocationID": "73cc4341603a47ca94a7a159ceb094b5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:31:32.902 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2335", "MemoryAccounting": "yes", "MemoryCurrent": "2805760", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:31:32.904 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:28:41 UTC", "StateChangeTimestampMonotonic": "11367120", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:28:39 UTC", "WatchdogTimestampMonotonic": "10316841", "WatchdogUSec": "0"}} 2020-03-11T22:31:32.939 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:32 +0000 (0:00:00.347) 0:02:09.507 ******* 2020-03-11T22:31:32.969 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:33.004 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.065) 0:02:09.572 ******* 2020-03-11T22:31:33.031 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:31:33.032 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:33.076 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.071) 0:02:09.644 ******* 2020-03-11T22:31:33.102 INFO:teuthology.task.ansible.out: 2020-03-11T22:31:33.103 INFO:teuthology.task.ansible.out:TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:33.139 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.063) 0:02:09.707 ******* 2020-03-11T22:31:33.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:31:33.345 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:31:33.399 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.259) 0:02:09.967 ******* 2020-03-11T22:31:33.880 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:31:33.881 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:31:33.920 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.521) 0:02:10.488 ******* 2020-03-11T22:31:35.180 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:31:35.181 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:31:35.219 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:01.299) 0:02:11.787 ******* 2020-03-11T22:31:35.245 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:31:35.246 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:35.282 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.062) 0:02:11.849 ******* 2020-03-11T22:31:35.318 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:31:35.320 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-11T22:31:35.368 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.086) 0:02:11.936 ******* 2020-03-11T22:31:35.559 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:31:35.560 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:31:35.595 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.226) 0:02:12.162 ******* 2020-03-11T22:31:35.621 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:31:35.622 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:35.657 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.062) 0:02:12.225 ******* 2020-03-11T22:31:35.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:31:35.687 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:31:35.721 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.063) 0:02:12.289 ******* 2020-03-11T22:31:35.747 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:31:35.749 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:35.784 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:00.062) 0:02:12.351 ******* 2020-03-11T22:31:35.985 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:31:35.986 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:31:36.021 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.237) 0:02:12.589 ******* 2020-03-11T22:31:36.213 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:31:36.215 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:31:36.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.230) 0:02:12.819 ******* 2020-03-11T22:31:36.278 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:31:36.279 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.318 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.066) 0:02:12.886 ******* 2020-03-11T22:31:36.355 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:31:36.357 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.367 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.368 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.379 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.380 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.426 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.107) 0:02:12.993 ******* 2020-03-11T22:31:36.462 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:31:36.463 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.464 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.475 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.486 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.498 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.536 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.109) 0:02:13.103 ******* 2020-03-11T22:31:36.562 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.603 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.066) 0:02:13.170 ******* 2020-03-11T22:31:36.639 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:31:36.641 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.679 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.076) 0:02:13.246 ******* 2020-03-11T22:31:36.705 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:31:36.707 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.746 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.066) 0:02:13.313 ******* 2020-03-11T22:31:36.782 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:31:36.784 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.843 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.097) 0:02:13.411 ******* 2020-03-11T22:31:36.869 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:31:36.871 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.920 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.077) 0:02:13.488 ******* 2020-03-11T22:31:36.947 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:31:36.948 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:36.999 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:36 +0000 (0:00:00.078) 0:02:13.567 ******* 2020-03-11T22:31:37.028 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:31:37.030 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:37.074 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:37 +0000 (0:00:00.074) 0:02:13.641 ******* 2020-03-11T22:31:37.110 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:31:37.112 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:37.162 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:37 +0000 (0:00:00.088) 0:02:13.729 ******* 2020-03-11T22:31:37.396 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:31:37.396 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004836", "end": "2020-03-11 22:31:37.348476", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:31:37.343640", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:37.584 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002984", "end": "2020-03-11 22:31:37.543281", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:31:37.540297", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:37.771 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002866", "end": "2020-03-11 22:31:37.729909", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:31:37.727043", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:37.992 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.031515", "end": "2020-03-11 22:31:37.956197", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:31:37.924682", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:38.052 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:38 +0000 (0:00:00.889) 0:02:14.619 ******* 2020-03-11T22:31:38.081 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:31:38.140 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:38 +0000 (0:00:00.088) 0:02:14.708 ******* 2020-03-11T22:31:38.342 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:31:38.342 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:31:38.441 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-11T22:31:38.443 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-03-11T22:31:38.445 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:31:38.446 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi118.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:31:38 +0000 (0:00:00.305) 0:02:15.013 ******* =============================================================================== 2020-03-11T22:31:38.447 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 16.30s common : nrpe - Load SELinux policy package ---------------------------- 16.19s testnode : Zap all non-root disks --------------------------------------- 7.94s testnode : Install packages --------------------------------------------- 7.67s testnode : Base group workaround for BZ1782899 -------------------------- 5.43s Gathering Facts --------------------------------------------------------- 3.75s common : Install Kerberos Packages (RedHat) ----------------------------- 3.37s testnode : Ensure ceph dependency packages are not present. ------------- 3.06s testnode : Ensure ceph packages are not present. ------------------------ 2.50s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.97s testnode : Create logical volume(s) ------------------------------------- 1.97s testnode : Rebuild rpmdb ------------------------------------------------ 1.93s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.66s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.51s testnode : Configure local mirrorlists ---------------------------------- 1.49s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.44s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.39s testnode : Install packages via pip ------------------------------------- 1.30s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.08s common : Configure local epel mirrorlists ------------------------------- 1.05s 2020-03-11T22:31:38.755 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2020-03-11T22:31:38.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:31:38.759 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi118.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:31:38.827 INFO:teuthology.orchestra.run.smithi118:> true 2020-03-11T22:31:38.917 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:31:38.930 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:31:38.930 INFO:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:31:38.992 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:31:39.009 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:31:39.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2020-03-11T22:31:39.063 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2020-03-11T22:31:39.081 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:31:39.098 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:31:39.164 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2020-03-11T22:31:39.167 INFO:teuthology.orchestra.run.smithi118.stdout:210 Number of sources = 4 2020-03-11T22:31:39.167 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:31:39.168 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2020-03-11T22:31:39.168 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:39.168 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:39.168 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:39.169 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:39.169 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:31:39.189 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:31:39.189 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:31:39.190 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:31:39.190 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:31:39.205 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:31:39.205 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:31:39.205 INFO:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2020-03-11T22:31:39.243 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS 2020-03-11T22:31:39.244 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:39.244 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:31:39.244 INFO:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2020-03-11T22:31:39.318 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS 2020-03-11T22:31:39.320 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:39.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:39.557 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:31:39.558 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:31:39.751 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:31:39.751 INFO:teuthology.orchestra.run.smithi118:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:31:39.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:31:39.796 INFO:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:31:39.864 INFO:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:31:39.944 INFO:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:31:40.011 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2020-03-11T22:31:40.012 INFO:teuthology.orchestra.run.smithi118:> sudo yum clean all 2020-03-11T22:31:40.390 INFO:teuthology.orchestra.run.smithi118.stdout:239 files removed 2020-03-11T22:31:40.419 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2020-03-11T22:31:41.105 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 198 kB/s | 69 kB 00:00 2020-03-11T22:31:41.436 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 45 kB/s | 14 kB 00:00 2020-03-11T22:31:41.743 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 4.9 kB/s | 1.4 kB 00:00 2020-03-11T22:31:41.896 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-8 - AppStream 48 MB/s | 6.5 MB 00:00 2020-03-11T22:31:43.037 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-11T22:31:43.816 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:31:43.973 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for ceph-el8 owned by ktdreyer 336 kB/s | 38 kB 00:00 2020-03-11T22:31:44.336 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.0 MB 00:00 2020-03-11T22:31:45.589 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 5.2 MB/s | 30 kB 00:00 2020-03-11T22:31:46.574 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:31:46.576 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:31:46.576 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2020-03-11T22:31:46.576 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:31:46.577 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:31:46.578 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:31:46.579 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:31:46.580 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:31:46.581 INFO:teuthology.orchestra.run.smithi118.stdout:Install 21 Packages 2020-03-11T22:31:46.581 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2020-03-11T22:31:46.581 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:31:46.581 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 51 M 2020-03-11T22:31:46.581 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:31:47.429 INFO:teuthology.orchestra.run.smithi118.stdout:(1/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 11 MB/s | 9.8 MB 00:00 2020-03-11T22:31:47.472 INFO:teuthology.orchestra.run.smithi118.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 576 kB/s | 25 kB 00:00 2020-03-11T22:31:47.518 INFO:teuthology.orchestra.run.smithi118.stdout:(3/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 5.8 MB/s | 5.5 MB 00:00 2020-03-11T22:31:47.526 INFO:teuthology.orchestra.run.smithi118.stdout:(4/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 9.6 MB/s | 507 kB 00:00 2020-03-11T22:31:47.561 INFO:teuthology.orchestra.run.smithi118.stdout:(5/23): libradosstriper1-15.1.0-1941.g4acc112.e 8.4 MB/s | 377 kB 00:00 2020-03-11T22:31:47.625 INFO:teuthology.orchestra.run.smithi118.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 630 kB/s | 40 kB 00:00 2020-03-11T22:31:47.690 INFO:teuthology.orchestra.run.smithi118.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 748 kB/s | 49 kB 00:00 2020-03-11T22:31:47.762 INFO:teuthology.orchestra.run.smithi118.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 2.2 MB/s | 162 kB 00:00 2020-03-11T22:31:47.841 INFO:teuthology.orchestra.run.smithi118.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 4.3 MB/s | 352 kB 00:00 2020-03-11T22:31:47.886 INFO:teuthology.orchestra.run.smithi118.stdout:(10/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 17 MB/s | 6.2 MB 00:00 2020-03-11T22:31:47.893 INFO:teuthology.orchestra.run.smithi118.stdout:(11/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 5.8 MB/s | 309 kB 00:00 2020-03-11T22:31:47.922 INFO:teuthology.orchestra.run.smithi118.stdout:(12/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 3.1 MB/s | 113 kB 00:00 2020-03-11T22:31:47.933 INFO:teuthology.orchestra.run.smithi118.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 8.6 MB/s | 353 kB 00:00 2020-03-11T22:31:47.935 INFO:teuthology.orchestra.run.smithi118.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 34 MB/s | 44 kB 00:00 2020-03-11T22:31:47.936 INFO:teuthology.orchestra.run.smithi118.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 33 MB/s | 47 kB 00:00 2020-03-11T22:31:47.954 INFO:teuthology.orchestra.run.smithi118.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2020-03-11T22:31:48.029 INFO:teuthology.orchestra.run.smithi118.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 790 kB/s | 59 kB 00:00 2020-03-11T22:31:48.049 INFO:teuthology.orchestra.run.smithi118.stdout:(18/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-03-11T22:31:48.054 INFO:teuthology.orchestra.run.smithi118.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 34 MB/s | 179 kB 00:00 2020-03-11T22:31:48.067 INFO:teuthology.orchestra.run.smithi118.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2020-03-11T22:31:48.279 INFO:teuthology.orchestra.run.smithi118.stdout:(21/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 9.0 MB/s | 1.9 MB 00:00 2020-03-11T22:31:48.382 INFO:teuthology.orchestra.run.smithi118.stdout:(22/23): librados2-15.1.0-1941.g4acc112.el8.x86 11 MB/s | 3.6 MB 00:00 2020-03-11T22:31:49.318 INFO:teuthology.orchestra.run.smithi118.stdout:(23/23): ceph-common-15.1.0-1941.g4acc112.el8.x 7.5 MB/s | 21 MB 00:02 2020-03-11T22:31:49.321 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:49.321 INFO:teuthology.orchestra.run.smithi118.stdout:Total 18 MB/s | 51 MB 00:02 2020-03-11T22:31:49.327 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:31:49.369 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:31:49.370 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:31:49.748 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:31:49.748 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:31:50.805 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:31:50.980 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:31:51.162 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:31:51.560 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:31:51.737 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:31:51.816 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:31:52.037 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:31:52.162 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:31:52.387 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:31:52.562 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:31:52.609 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:31:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:31:52.953 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:31:53.584 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:31:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:31:53.828 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:31:53.946 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:31:54.114 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:31:54.298 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:31:54.503 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:31:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:31:54.684 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:31:54.857 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:31:55.086 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:31:55.228 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:31:55.409 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:31:55.537 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:31:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:31:59.194 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:31:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:32:00.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:32:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:32:00.369 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:32:16.464 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:32:16.464 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2020-03-11T22:32:16.464 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2020-03-11T22:32:16.464 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2020-03-11T22:32:16.464 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /run 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2020-03-11T22:32:16.465 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:32:17.948 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:32:17.949 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:32:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:32:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:32:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:32:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:32:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:32:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:32:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:32:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:32:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:32:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:32:19.597 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:19.597 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:19.597 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.599 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:32:19.600 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:32:19.601 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:32:19.601 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:32:19.601 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:32:19.601 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:32:19.601 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:19.602 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:19.894 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2020-03-11T22:32:20.301 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:35 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:20.699 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:20.700 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:20.700 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2020-03-11T22:32:20.700 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:20.700 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:32:20.700 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:32:20.701 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:20.702 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:32:20.702 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:20.702 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2020-03-11T22:32:20.702 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:20.703 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 43 M 2020-03-11T22:32:20.703 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 184 M 2020-03-11T22:32:20.703 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:32:20.735 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2020-03-11T22:32:20.743 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): socat-1.7.3.2-6.el8.x86_64.rpm 37 MB/s | 298 kB 00:00 2020-03-11T22:32:20.746 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.2 MB/s | 188 kB 00:00 2020-03-11T22:32:20.747 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 59 MB/s | 249 kB 00:00 2020-03-11T22:32:20.808 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-11T22:32:22.726 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 21 MB/s | 42 MB 00:02 2020-03-11T22:32:22.729 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:22.729 INFO:teuthology.orchestra.run.smithi118.stdout:Total 21 MB/s | 43 MB 00:02 2020-03-11T22:32:22.735 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:32:22.756 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:32:22.756 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:32:23.023 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:32:23.023 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:32:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:32:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:32:24.745 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:32:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:32:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:32:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:32:29.171 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:32:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:32:30.876 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:32:30.876 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:32:30.876 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:32:30.876 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:32:30.877 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:32:30.877 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:32:31.363 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:31.364 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:31.563 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2020-03-11T22:32:31.958 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:46 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:32.385 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:32.389 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:32.389 INFO:teuthology.orchestra.run.smithi118.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:32.389 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:32:32.390 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:32:32.391 INFO:teuthology.orchestra.run.smithi118.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:32:32.392 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:32:32.393 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:32:32.394 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout:Install 36 Packages 2020-03-11T22:32:32.395 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:32.396 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 29 M 2020-03-11T22:32:32.397 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 105 M 2020-03-11T22:32:32.397 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 41 kB/s | 6.4 kB 00:00 2020-03-11T22:32:32.770 INFO:teuthology.orchestra.run.smithi118.stdout:(2/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 5.4 MB/s | 2.0 MB 00:00 2020-03-11T22:32:32.860 INFO:teuthology.orchestra.run.smithi118.stdout:(3/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.1 MB/s | 1.4 MB 00:00 2020-03-11T22:32:32.927 INFO:teuthology.orchestra.run.smithi118.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.2 MB/s | 165 kB 00:00 2020-03-11T22:32:32.961 INFO:teuthology.orchestra.run.smithi118.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.2 MB/s | 214 kB 00:00 2020-03-11T22:32:32.981 INFO:teuthology.orchestra.run.smithi118.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 80 MB/s | 1.5 MB 00:00 2020-03-11T22:32:33.001 INFO:teuthology.orchestra.run.smithi118.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 60 MB/s | 157 kB 00:00 2020-03-11T22:32:33.002 INFO:teuthology.orchestra.run.smithi118.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2020-03-11T22:32:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2020-03-11T22:32:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2020-03-11T22:32:33.004 INFO:teuthology.orchestra.run.smithi118.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 74 MB/s | 457 kB 00:00 2020-03-11T22:32:33.004 INFO:teuthology.orchestra.run.smithi118.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 60 MB/s | 195 kB 00:00 2020-03-11T22:32:33.004 INFO:teuthology.orchestra.run.smithi118.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2020-03-11T22:32:33.005 INFO:teuthology.orchestra.run.smithi118.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 51 MB/s | 123 kB 00:00 2020-03-11T22:32:33.007 INFO:teuthology.orchestra.run.smithi118.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 61 MB/s | 176 kB 00:00 2020-03-11T22:32:33.064 INFO:teuthology.orchestra.run.smithi118.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.4 MB/s | 145 kB 00:00 2020-03-11T22:32:33.093 INFO:teuthology.orchestra.run.smithi118.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 18 MB/s | 385 kB 00:00 2020-03-11T22:32:33.095 INFO:teuthology.orchestra.run.smithi118.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.1 MB/s | 11 kB 00:00 2020-03-11T22:32:33.103 INFO:teuthology.orchestra.run.smithi118.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.1 MB/s | 18 kB 00:00 2020-03-11T22:32:33.113 INFO:teuthology.orchestra.run.smithi118.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 4.5 MB/s | 49 kB 00:00 2020-03-11T22:32:33.124 INFO:teuthology.orchestra.run.smithi118.stdout:(21/36): python3-more-itertools-7.2.0-3.el8.noa 6.4 MB/s | 59 kB 00:00 2020-03-11T22:32:33.146 INFO:teuthology.orchestra.run.smithi118.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 24 MB/s | 283 kB 00:00 2020-03-11T22:32:33.149 INFO:teuthology.orchestra.run.smithi118.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 2.0 MB/s | 16 kB 00:00 2020-03-11T22:32:33.152 INFO:teuthology.orchestra.run.smithi118.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.3 MB/s | 19 kB 00:00 2020-03-11T22:32:33.160 INFO:teuthology.orchestra.run.smithi118.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-11T22:32:33.168 INFO:teuthology.orchestra.run.smithi118.stdout:(26/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.5 MB/s | 30 kB 00:00 2020-03-11T22:32:33.178 INFO:teuthology.orchestra.run.smithi118.stdout:(27/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.1 MB/s | 27 kB 00:00 2020-03-11T22:32:33.197 INFO:teuthology.orchestra.run.smithi118.stdout:(28/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.7 MB/s | 23 kB 00:00 2020-03-11T22:32:33.248 INFO:teuthology.orchestra.run.smithi118.stdout:(29/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 705 kB/s | 44 kB 00:00 2020-03-11T22:32:33.294 INFO:teuthology.orchestra.run.smithi118.stdout:(30/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.9 MB/s | 185 kB 00:00 2020-03-11T22:32:33.310 INFO:teuthology.orchestra.run.smithi118.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.5 MB/s | 40 kB 00:00 2020-03-11T22:32:33.341 INFO:teuthology.orchestra.run.smithi118.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 7.6 MB/s | 241 kB 00:00 2020-03-11T22:32:33.358 INFO:teuthology.orchestra.run.smithi118.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 14 MB/s | 251 kB 00:00 2020-03-11T22:32:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 5.2 MB/s | 85 kB 00:00 2020-03-11T22:32:33.542 INFO:teuthology.orchestra.run.smithi118.stdout:(35/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 22 MB/s | 17 MB 00:00 2020-03-11T22:32:33.801 INFO:teuthology.orchestra.run.smithi118.stdout:(36/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 3.5 MB/s | 4.4 MB 00:01 2020-03-11T22:32:33.804 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:33.804 INFO:teuthology.orchestra.run.smithi118.stdout:Total 21 MB/s | 29 MB 00:01 2020-03-11T22:32:33.823 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:32:33.861 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:32:33.861 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:32:34.210 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:32:34.210 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:32:35.496 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:32:35.691 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:32:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:32:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:32:36.297 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:32:36.487 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:32:36.655 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:32:36.828 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:32:36.967 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:32:37.150 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:32:37.291 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:32:37.426 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:32:37.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:32:37.894 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:32:38.087 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:32:38.336 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:32:38.559 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:32:38.814 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:32:39.019 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:32:39.280 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:32:39.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:32:39.871 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:32:40.122 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:32:40.318 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:32:40.523 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:32:40.679 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:32:40.902 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:32:41.150 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:32:41.390 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:32:41.739 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:32:42.010 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:32:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:32:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:32:43.953 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:32:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:32:44.482 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:32:44.603 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:32:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:32:44.928 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:32:44.946 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:32:45.060 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:32:45.400 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:32:45.400 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:32:45.400 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:32:45.400 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:32:45.401 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:32:45.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:32:45.403 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:32:45.403 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:32:45.403 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:32:45.404 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:32:45.405 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:32:45.405 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:32:45.405 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:32:45.405 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:32:45.405 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:32:45.406 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:32:45.406 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:32:45.406 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:32:45.406 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:32:45.406 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:32:46.068 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:32:46.069 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:46.069 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:46.069 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:46.069 INFO:teuthology.orchestra.run.smithi118.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:32:46.070 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:32:46.070 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:46.070 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:46.070 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:46.070 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:32:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:32:46.072 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:32:46.073 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:32:46.073 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:32:46.073 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:32:46.073 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:32:46.074 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:32:46.075 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:32:46.075 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:32:46.075 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:32:46.075 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:32:46.075 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:32:46.076 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:32:46.076 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:32:46.076 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:46.076 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:46.430 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2020-03-11T22:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:47.196 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:47.197 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:32:47.197 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:47.354 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2020-03-11T22:32:47.746 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:48.119 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:48.120 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:48.120 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2020-03-11T22:32:48.120 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:48.120 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:48.121 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 36 k 2020-03-11T22:32:48.122 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 129 k 2020-03-11T22:32:48.122 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:32:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 189 kB/s | 36 kB 00:00 2020-03-11T22:32:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:Total 188 kB/s | 36 kB 00:00 2020-03-11T22:32:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:32:48.319 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:32:48.319 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:32:48.320 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:32:48.320 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:32:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:32:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:50.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:50.167 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:50.167 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:50.168 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:50.168 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:50.168 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:50.323 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:32:50.724 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:51.099 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:51.100 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:32:51.101 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:51.101 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:32:51.101 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:51.101 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 203 k 2020-03-11T22:32:51.102 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 743 k 2020-03-11T22:32:51.102 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:32:51.359 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 787 kB/s | 203 kB 00:00 2020-03-11T22:32:51.359 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:51.360 INFO:teuthology.orchestra.run.smithi118.stdout:Total 784 kB/s | 203 kB 00:00 2020-03-11T22:32:51.362 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:32:51.374 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:32:51.374 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:32:51.419 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:32:51.420 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:32:51.542 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:32:51.694 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:52.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:52.391 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:52.391 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:52.392 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:52.392 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:52.392 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:52.688 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2020-03-11T22:32:53.089 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:08 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:53.371 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:53.477 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:53.478 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:32:53.479 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:53.636 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:32:54.031 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:09 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:32:54.402 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:32:54.402 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:32:54.403 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:32:54.404 INFO:teuthology.orchestra.run.smithi118.stdout:Install 5 Packages 2020-03-11T22:32:54.405 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:54.405 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 4.2 M 2020-03-11T22:32:54.405 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 19 M 2020-03-11T22:32:54.406 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:32:54.435 INFO:teuthology.orchestra.run.smithi118.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2020-03-11T22:32:54.484 INFO:teuthology.orchestra.run.smithi118.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 698 kB/s | 34 kB 00:00 2020-03-11T22:32:54.509 INFO:teuthology.orchestra.run.smithi118.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 7.6 MB/s | 197 kB 00:00 2020-03-11T22:32:54.594 INFO:teuthology.orchestra.run.smithi118.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 118 kB/s | 22 kB 00:00 2020-03-11T22:32:54.911 INFO:teuthology.orchestra.run.smithi118.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 7.6 MB/s | 3.9 MB 00:00 2020-03-11T22:32:54.913 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:54.914 INFO:teuthology.orchestra.run.smithi118.stdout:Total 8.1 MB/s | 4.2 MB 00:00 2020-03-11T22:32:54.918 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:32:54.926 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:32:54.926 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:32:55.070 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:32:55.070 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:32:55.303 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:32:55.541 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:32:55.769 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:32:55.950 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:32:56.937 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:32:57.210 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:32:57.409 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:32:57.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:32:57.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:32:57.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:32:57.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:32:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:32:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:58.983 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:32:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:32:58.984 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:32:59.751 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:33:00.153 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:00.518 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:00.519 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:00.519 INFO:teuthology.orchestra.run.smithi118.stdout: Package ArchVersion Repository Size 2020-03-11T22:33:00.519 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:00.519 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:33:00.520 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout:Install 8 Packages 2020-03-11T22:33:00.521 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:00.522 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2020-03-11T22:33:00.522 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 205 M 2020-03-11T22:33:00.522 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:00.623 INFO:teuthology.orchestra.run.smithi118.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 43 MB/s | 4.3 MB 00:00 2020-03-11T22:33:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 60 MB/s | 3.7 MB 00:00 2020-03-11T22:33:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:(3/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2020-03-11T22:33:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 16 MB/s | 225 kB 00:00 2020-03-11T22:33:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 45 MB/s | 638 kB 00:00 2020-03-11T22:33:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 31 MB/s | 169 kB 00:00 2020-03-11T22:33:00.962 INFO:teuthology.orchestra.run.smithi118.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 55 MB/s | 15 MB 00:00 2020-03-11T22:33:01.285 INFO:teuthology.orchestra.run.smithi118.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 11 MB/s | 8.6 MB 00:00 2020-03-11T22:33:01.288 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:01.288 INFO:teuthology.orchestra.run.smithi118.stdout:Total 48 MB/s | 37 MB 00:00 2020-03-11T22:33:01.299 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:01.317 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:01.318 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:01.616 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:01.617 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:02.156 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:02.300 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:33:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:33:02.567 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:33:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:33:03.246 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:33:03.781 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:33:03.942 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:33:04.642 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:33:04.954 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:33:05.158 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:33:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:33:08.407 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:33:08.574 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:33:08.843 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:33:08.844 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:33:08.845 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:33:09.123 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:09.124 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:09.298 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:33:09.694 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:10.058 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:33:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-noarch 24 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:33:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:10.062 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:10.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:10.063 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2020-03-11T22:33:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:10.063 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 1.9 M 2020-03-11T22:33:10.064 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 23 M 2020-03-11T22:33:10.064 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:10.093 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 873 kB/s | 27 kB 00:00 2020-03-11T22:33:10.097 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 35 MB/s | 126 kB 00:00 2020-03-11T22:33:10.097 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 580 kB/s | 20 kB 00:00 2020-03-11T22:33:10.100 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 35 MB/s | 110 kB 00:00 2020-03-11T22:33:10.101 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2020-03-11T22:33:10.102 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 31 MB/s | 43 kB 00:00 2020-03-11T22:33:10.148 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 703 kB/s | 33 kB 00:00 2020-03-11T22:33:10.164 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 261 kB/s | 16 kB 00:00 2020-03-11T22:33:10.179 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-03-11T22:33:10.196 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.4 MB/s | 73 kB 00:00 2020-03-11T22:33:10.211 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.7 MB/s | 61 kB 00:00 2020-03-11T22:33:10.297 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-kubernetes-10.0.1-2.el8.noarch 8.4 MB/s | 1.1 MB 00:00 2020-03-11T22:33:10.456 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 62 kB/s | 24 kB 00:00 2020-03-11T22:33:10.458 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:10.458 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.7 MB/s | 1.9 MB 00:00 2020-03-11T22:33:10.465 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:10.479 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:10.479 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:10.544 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:10.544 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:11.023 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:33:11.487 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:33:11.914 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:33:12.113 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:33:12.294 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:33:12.560 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:33:12.803 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:33:13.848 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:33:14.080 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:33:14.256 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:33:14.439 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:33:14.607 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:33:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:33:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:33:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:33:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:15.930 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:33:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:15.932 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:16.134 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:33:16.530 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:16.906 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:16.907 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: Package ArchVersion Repository Size 2020-03-11T22:33:16.907 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:16.909 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:16.909 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:16.909 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2020-03-11T22:33:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:16.910 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 461 k 2020-03-11T22:33:16.910 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.4 M 2020-03-11T22:33:16.910 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:16.963 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 901 kB/s | 48 kB 00:00 2020-03-11T22:33:16.982 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 4.8 MB/s | 356 kB 00:00 2020-03-11T22:33:17.010 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 456 kB/s | 21 kB 00:00 2020-03-11T22:33:17.073 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 221 kB/s | 36 kB 00:00 2020-03-11T22:33:17.074 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:17.075 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.7 MB/s | 461 kB 00:00 2020-03-11T22:33:17.079 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:17.085 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:17.085 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:17.115 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:17.115 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:17.299 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:17.541 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:33:17.789 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:33:17.987 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:33:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:33:18.341 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:33:18.341 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:33:18.342 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:33:18.342 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:33:18.579 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:33:18.579 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:18.579 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:18.580 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:18.765 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2020-03-11T22:33:19.173 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:19.538 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:19.539 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2020-03-11T22:33:19.540 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout:Install 3 Packages 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:19.541 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 676 k 2020-03-11T22:33:19.542 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.1 M 2020-03-11T22:33:19.542 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:19.572 INFO:teuthology.orchestra.run.smithi118.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-11T22:33:19.575 INFO:teuthology.orchestra.run.smithi118.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 617 kB/s | 21 kB 00:00 2020-03-11T22:33:20.051 INFO:teuthology.orchestra.run.smithi118.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.1 MB/s | 572 kB 00:00 2020-03-11T22:33:20.052 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:20.053 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.3 MB/s | 676 kB 00:00 2020-03-11T22:33:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:20.080 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:20.081 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:20.142 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:20.142 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:20.305 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:20.306 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:33:20.306 INFO:teuthology.orchestra.run.smithi118.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:33:20.306 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:33:20.929 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:33:21.567 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:33:21.568 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:33:21.568 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:33:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:33:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:21.854 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:33:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:33:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:21.855 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:22.159 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2020-03-11T22:33:22.567 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:22.974 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:22.975 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2020-03-11T22:33:22.975 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:33:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:22.977 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 141 k 2020-03-11T22:33:22.977 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 487 k 2020-03-11T22:33:22.977 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:23.509 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 264 kB/s | 141 kB 00:00 2020-03-11T22:33:23.510 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:23.510 INFO:teuthology.orchestra.run.smithi118.stdout:Total 263 kB/s | 141 kB 00:00 2020-03-11T22:33:23.512 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:23.519 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:23.520 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:23.570 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:23.570 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:24.717 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2020-03-11T22:33:25.122 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:25.407 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:25.516 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:25.517 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:25.517 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:25.722 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2020-03-11T22:33:26.121 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:26.506 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:26.507 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2020-03-11T22:33:26.507 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:26.507 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:33:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:26.508 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:26.508 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:26.508 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:33:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:26.509 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 21 k 2020-03-11T22:33:26.509 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 76 k 2020-03-11T22:33:26.509 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:26.816 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 69 kB/s | 21 kB 00:00 2020-03-11T22:33:26.817 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:26.817 INFO:teuthology.orchestra.run.smithi118.stdout:Total 69 kB/s | 21 kB 00:00 2020-03-11T22:33:26.819 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:26.823 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:26.824 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:26.826 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:26.826 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:27.052 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.229 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.436 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.436 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:27.436 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:27.436 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:27.437 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:27.437 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:27.655 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2020-03-11T22:33:28.052 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:28.323 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:28.429 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:28.430 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:28.431 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:28.632 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2020-03-11T22:33:29.028 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:29.323 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:29.429 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:29.430 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:29.431 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:29.629 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2020-03-11T22:33:30.028 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:30.314 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:30.420 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:30.421 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:30.421 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:30.632 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2020-03-11T22:33:31.029 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:31.323 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:31.429 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:31.430 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:31.430 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:31.636 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2020-03-11T22:33:32.042 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:32.322 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:32.427 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:32.428 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:32.429 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:32.620 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2020-03-11T22:33:33.019 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:33.297 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:33.406 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:33.407 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2020-03-11T22:33:33.407 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:33.581 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2020-03-11T22:33:33.977 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:34.338 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:33:34.339 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 72 k 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 230 k 2020-03-11T22:33:34.340 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:34.536 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 369 kB/s | 72 kB 00:00 2020-03-11T22:33:34.536 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:34.536 INFO:teuthology.orchestra.run.smithi118.stdout:Total 367 kB/s | 72 kB 00:00 2020-03-11T22:33:34.538 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:34.549 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:34.549 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:34.597 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:34.597 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:34.700 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:34.877 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:35.108 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:35.344 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:35.345 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:35.672 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2020-03-11T22:33:36.076 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:36.511 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:36.512 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:36.512 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2020-03-11T22:33:36.512 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:36.512 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:36.513 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.7 M 2020-03-11T22:33:36.514 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 12 M 2020-03-11T22:33:36.514 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 5.1 MB/s | 2.7 MB 00:00 2020-03-11T22:33:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:37.042 INFO:teuthology.orchestra.run.smithi118.stdout:Total 5.1 MB/s | 2.7 MB 00:00 2020-03-11T22:33:37.044 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:37.057 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:37.057 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:37.116 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:37.116 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:37.826 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:38.812 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:39.130 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:39.378 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:39.378 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:39.378 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:39.379 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:39.379 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:39.379 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:39.713 INFO:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2020-03-11T22:33:40.115 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:55 ago on Wed 11 Mar 2020 10:31:45 PM UTC. 2020-03-11T22:33:40.513 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:33:40.514 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 141 k 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 526 k 2020-03-11T22:33:40.515 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2020-03-11T22:33:40.741 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 624 kB/s | 141 kB 00:00 2020-03-11T22:33:40.741 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:40.741 INFO:teuthology.orchestra.run.smithi118.stdout:Total 621 kB/s | 141 kB 00:00 2020-03-11T22:33:40.743 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2020-03-11T22:33:40.754 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2020-03-11T22:33:40.754 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2020-03-11T22:33:40.802 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2020-03-11T22:33:40.802 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2020-03-11T22:33:40.917 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2020-03-11T22:33:41.134 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:41.322 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:41.559 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2020-03-11T22:33:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: 2020-03-11T22:33:41.560 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2020-03-11T22:33:41.896 DEBUG:teuthology.parallel:result is None 2020-03-11T22:33:41.896 INFO:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2020-03-11T22:33:41.933 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS 2020-03-11T22:33:41.934 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:33:41.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:42.151 INFO:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:33:42.169 INFO:teuthology.orchestra.run.smithi118.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:33:42.170 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:33:42.170 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:33:42.170 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:33:42.176 INFO:teuthology.orchestra.run.smithi118:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:33:42.239 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:33:42.242 INFO:teuthology.orchestra.run.smithi118:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:33:42.307 INFO:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:33:42.374 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:33:42.375 INFO:teuthology.orchestra.run.smithi118:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:33:42.463 INFO:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:33:42.531 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:33:42.550 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:42.550 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:33:42.550 INFO:tasks.workunit:timeout=3h 2020-03-11T22:33:42.551 INFO:tasks.workunit:cleanup=True 2020-03-11T22:33:42.551 INFO:teuthology.orchestra.run.smithi118:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:42.585 INFO:teuthology.orchestra.run.smithi118.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:33:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:33:42.586 INFO:teuthology.orchestra.run.smithi118:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:42.639 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:42.640 INFO:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:33:42.694 INFO:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:42.804 INFO:tasks.workunit.client.0.smithi118.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:35:17.591 INFO:tasks.workunit.client.0.smithi118.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:35:17.591 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:35:17.591 INFO:tasks.workunit.client.0.smithi118.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:35:17.591 INFO:tasks.workunit.client.0.smithi118.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:35:17.591 INFO:tasks.workunit.client.0.smithi118.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr: git checkout -b 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:35:17.592 INFO:tasks.workunit.client.0.smithi118.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:35:17.596 INFO:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:35:17.667 DEBUG:teuthology.orchestra.remote:smithi118:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:35:17.715 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:35:17.715 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:35:17.716 INFO:teuthology.orchestra.run.smithi118:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:35:17.741 INFO:tasks.workunit.client.0.smithi118.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:35:17.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:35:17.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:35:17.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:35:17.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:35:17.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:35:17.755 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:35:17.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:35:17.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:35:17.756 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:35:17.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:35:17.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:35:17.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:35:17.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:35:17.758 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:35:17.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none ' 2020-03-11T22:35:17.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:35:17.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:35:17.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:35:17.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:17.763 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:35:17.763 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:35:17.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:35:17.765 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_divergent_2 2020-03-11T22:35:17.765 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_divergent_3 2020-03-11T22:35:17.765 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_divergent_ec' 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:17.767 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:17.767 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:17.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:17.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:17.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:17.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:17.768 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:17.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:17.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:17.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:17.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:17.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:17.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:17.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:17.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:17.770 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:17.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:17.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:17.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:17.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:17.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:17.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:17.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:35:17.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:17.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:17.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:35:17.774 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:35:17.775 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:17.775 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:17.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17426 2020-03-11T22:35:17.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:35:17.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:35:17.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:35:17.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:35:17.778 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:35:17.778 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:35:17.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:35:17.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:35:17.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:35:17.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:35:17.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:35:18.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:35:18.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:18.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:35:18.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:35:18.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:35:18.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:18.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:18.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:18.290 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:35:18.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:18.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get fsid 2020-03-11T22:35:18.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:35:18.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:18.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:18.377 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:18.378 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:18.378 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:18.378 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:35:18.378 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:18.378 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get mon_host 2020-03-11T22:35:18.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:35:18.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:35:18.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:35:18.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:35:18.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:35:18.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:35:18.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:35:19.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:35:19.528 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:19.528 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:19.528 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:19.528 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:19.529 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:19.529 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:19.529 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:19.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:19.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:19.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:35:19.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:19.558 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:19.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:35:19.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:19.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:35:19.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 a11e7915-ce48-4dfd-9e2b-369bb6b4c050' 2020-03-11T22:35:19.561 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:19.561 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:35:19.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCnZ2leXXgMIhAAIDPaNsZZMxLYqY2xLNgrxQ== 2020-03-11T22:35:19.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCnZ2leXXgMIhAAIDPaNsZZMxLYqY2xLNgrxQ=="}' 2020-03-11T22:35:19.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a11e7915-ce48-4dfd-9e2b-369bb6b4c050 -i td/divergent-priors/0/new.json 2020-03-11T22:35:19.919 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:35:19.928 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:35:19.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCnZ2leXXgMIhAAIDPaNsZZMxLYqY2xLNgrxQ== --osd-uuid a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:35:19.958 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.955+0000 7fdff2132ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:19.959 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.957+0000 7fdff2132ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:19.960 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.958+0000 7fdff2132ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:19.961 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.958+0000 7fdff2132ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:19.961 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.958+0000 7fdff2132ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:19.961 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:19.958+0000 7fdff2132ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:20.003 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:20.001+0000 7fdff2132ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:35:22.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:35:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:35:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:22.951 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:35:23.216 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:35:23.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:35:23.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:23.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:23.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:23.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:23.243 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:23.241+0000 7f8170341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.266 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:23.263+0000 7f8170341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.267 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:23.265+0000 7f8170341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:35:23.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:23.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:23.505 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:35:23.782 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:23.779+0000 7f8170341ec0 -1 Falling back to public interface 2020-03-11T22:35:23.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:24.523 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:24.520+0000 7f8170341ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:35:24.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:24.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:24.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:24.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:24.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:24.871 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:35:25.242 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17808,v1:127.0.0.1:6803/17808] [v2:127.0.0.1:6804/17808,v1:127.0.0.1:6805/17808] exists,up a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:35:25.242 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:25.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:25.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:25.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:25.246 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:25.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:25.248 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:25.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:35:25.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 d70199f6-ee80-4d1e-87ec-b3fe4e92a1de' 2020-03-11T22:35:25.249 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:25.259 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCtZ2leGGdSDxAAIm5X+WfLYCfSb/XkkzidRg== 2020-03-11T22:35:25.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCtZ2leGGdSDxAAIm5X+WfLYCfSb/XkkzidRg=="}' 2020-03-11T22:35:25.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d70199f6-ee80-4d1e-87ec-b3fe4e92a1de -i td/divergent-priors/1/new.json 2020-03-11T22:35:25.784 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:35:25.794 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:35:25.795 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCtZ2leGGdSDxAAIm5X+WfLYCfSb/XkkzidRg== --osd-uuid d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:35:25.823 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.820+0000 7f3d19436ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.824 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.822+0000 7f3d19436ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.826 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.823+0000 7f3d19436ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.826 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.824+0000 7f3d19436ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:25.827 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.824+0000 7f3d19436ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:25.827 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.824+0000 7f3d19436ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:25.897 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:25.894+0000 7f3d19436ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:35:28.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:35:28.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:28.883 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:35:28.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:35:28.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:29.336 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:35:29.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:35:29.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:29.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:29.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:29.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:29.363 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:29.360+0000 7f46eda16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.364 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:29.362+0000 7f46eda16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.365 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:29.363+0000 7f46eda16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:29.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:29.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:29.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:29.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:29.683 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:35:29.882 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:29.879+0000 7f46eda16ec0 -1 Falling back to public interface 2020-03-11T22:35:30.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:30.638 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:30.636+0000 7f46eda16ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:35:31.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:31.127 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:31.127 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:31.127 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:35:31.128 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:31.128 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:31.448 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18218,v1:127.0.0.1:6811/18218] [v2:127.0.0.1:6812/18218,v1:127.0.0.1:6813/18218] exists,up d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:35:31.448 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:31.448 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:31.448 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:31.448 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:31.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:35:31.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:31.451 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:31.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:31.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:31.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:31.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:35:31.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:31.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:35:31.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 e5f169f3-9161-49aa-ac04-828defbd3e0d' 2020-03-11T22:35:31.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:31.455 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:35:31.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCzZ2leQcy3GxAA2vW63E0DJFpWBVg6vHxh8g== 2020-03-11T22:35:31.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCzZ2leQcy3GxAA2vW63E0DJFpWBVg6vHxh8g=="}' 2020-03-11T22:35:31.468 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e5f169f3-9161-49aa-ac04-828defbd3e0d -i td/divergent-priors/2/new.json 2020-03-11T22:35:31.960 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:35:31.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:35:31.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCzZ2leQcy3GxAA2vW63E0DJFpWBVg6vHxh8g== --osd-uuid e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:35:31.998 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:31.996+0000 7f8e924dbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:32.000 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:31.997+0000 7f8e924dbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:32.002 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:31.999+0000 7f8e924dbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:32.002 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:32.000+0000 7f8e924dbec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:32.003 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:32.000+0000 7f8e924dbec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:32.003 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:32.000+0000 7f8e924dbec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:32.071 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:32.068+0000 7f8e924dbec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:35:35.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:35.082 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:35:35.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:35:35.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:35.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:35.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:35.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:35.538 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:35:35.563 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:35.559+0000 7f5c67278ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:35.564 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:35.561+0000 7f5c67278ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:35.566 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:35.563+0000 7f5c67278ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:35.900 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:35:35.900 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:35.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:35.902 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:35:36.083 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:36.078+0000 7f5c67278ec0 -1 Falling back to public interface 2020-03-11T22:35:36.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:36.855 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:35:36.851+0000 7f5c67278ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:35:37.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:37.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:37.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:37.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:37.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:37.244 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:35:37.608 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18591,v1:127.0.0.1:6819/18591] [v2:127.0.0.1:6820/18591,v1:127.0.0.1:6821/18591] exists,up e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:35:37.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:37.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:37.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:37.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:35:38.223 INFO:tasks.workunit.client.0.smithi118.stderr:noout is set 2020-03-11T22:35:38.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:35:38.942 INFO:tasks.workunit.client.0.smithi118.stderr:noin is set 2020-03-11T22:35:38.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:35:39.652 INFO:tasks.workunit.client.0.smithi118.stderr:nodown is set 2020-03-11T22:35:39.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:35:39.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:35:40.126 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' already exists 2020-03-11T22:35:40.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:35:41.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:35:41.783 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 3 2020-03-11T22:35:41.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:35:42.457 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 min_size to 2 2020-03-11T22:35:42.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:35:42.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:42.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:42.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:42.809 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:35:42.809 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:35:42.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:42.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:42.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:42.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:35:42.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:35:42.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:42.939 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:43.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:35:43.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:35:43.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:43.070 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:43.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509443 2020-03-11T22:35:43.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509443' 2020-03-11T22:35:43.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:43.198 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:43.198 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:43.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:43.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:43.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:43.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:35:43.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:35:43.201 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:35:43.202 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:43.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:35:43.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:43.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:35:43.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:43.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:43.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:35:43.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:43.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:35:43.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:35:43.558 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:35:43.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:43.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:35:43.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:43.909 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509443 2020-03-11T22:35:43.910 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:43.910 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:43.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509443 2020-03-11T22:35:43.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:43.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509443 2020-03-11T22:35:43.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2020-03-11T22:35:43.913 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509443 2020-03-11T22:35:43.913 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509443 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:44.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:44.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:44.345 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:44.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:44.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:44.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:44.346 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:44.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:44.687 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:35:44.687 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:35:44.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:44.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:44.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:44.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836488 2020-03-11T22:35:44.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488' 2020-03-11T22:35:44.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:44.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:44.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:35:44.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2020-03-11T22:35:44.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:44.947 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:45.075 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:35:45.075 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509446' 2020-03-11T22:35:45.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:45.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836488 2020-03-11T22:35:45.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:45.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:45.078 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836488 2020-03-11T22:35:45.078 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:45.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836488 2020-03-11T22:35:45.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-03-11T22:35:45.079 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836488 2020-03-11T22:35:45.080 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:45.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836488 2020-03-11T22:35:45.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:46.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:46.419 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:46.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836488 2020-03-11T22:35:46.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:46.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:35:46.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:46.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:46.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:35:46.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:46.772 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:35:46.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:35:46.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:35:46.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:47.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672967 -lt 42949672967 2020-03-11T22:35:47.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:47.118 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:47.118 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:47.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:47.120 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:47.120 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:47.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:35:47.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:35:47.121 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:35:47.122 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:47.525 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:35:47.526 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:35:47.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:47.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:47.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:35:47.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:47.969 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:47.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:47.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:47.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:47.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:47.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:48.301 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:35:48.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:48.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:48.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:48.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:35:48.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:35:48.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:35:48.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:35:48.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:35:49.104 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:35:49.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:35:49.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:35:49.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:49.113 INFO:tasks.workunit.client.0.smithi118.stdout:primary and soon to be divergent is 1 2020-03-11T22:35:49.431 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:49.431 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:35:41.625033+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:35:49.432 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:35:49.432 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:49.433 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:35:49.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:35:49.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:49.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:35:49.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:35:49.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:35:49.443 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:35:49.444 INFO:tasks.workunit.client.0.smithi118.stdout:writing initial objects 2020-03-11T22:35:49.444 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.444 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:35:49.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:35:49.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:35:49.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:35:49.681 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:35:49.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:35:49.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:35:49.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:35:49.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:49.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:35:50.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:50.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:35:50.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:50.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:35:50.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:50.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:35:50.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:50.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:35:50.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:50.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:35:51.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:51.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:35:51.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:51.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:35:51.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:51.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:35:52.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:35:52.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:35:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:35:52.781 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.781 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:35:52.832 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.832 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:35:52.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:35:52.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:35:52.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:52.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:35:53.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:35:53.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:35:53.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:35:53.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:35:53.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:35:53.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:35:53.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:35:53.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:35:53.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:35:53.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:35:53.641 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.641 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:35:53.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:35:53.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:35:53.815 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.815 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:35:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:35:53.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:35:53.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:53.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:35:54.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:35:54.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:35:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:35:54.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:35:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:35:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:35:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:35:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:35:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:35:54.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:35:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:35:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:35:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:35:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:35:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:35:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:35:55.090 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.090 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:35:55.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:35:55.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:35:55.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:35:55.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:35:55.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:35:55.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:35:55.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:35:55.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:35:55.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:35:55.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:35:55.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:35:55.840 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.840 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:35:55.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:35:55.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:55.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:35:56.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:35:56.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:35:56.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:35:56.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:35:56.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:35:56.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:35:56.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:35:56.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:35:56.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:35:56.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:35:56.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:35:56.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:35:56.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:35:56.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:35:56.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:35:56.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:35:56.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:56.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:35:57.005 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.005 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:35:57.044 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:35:57.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.106 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:35:57.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:35:57.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:35:57.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:35:57.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:35:57.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:35:57.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:57.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:35:57.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:35:57.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:35:57.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:57.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:57.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:57.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:57.530 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:35:57.530 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:57.530 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:57.530 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:57.531 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:57.531 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:57.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:57.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:57.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:57.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:57.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:57.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:57.929 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:35:57.929 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:35:57.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:57.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:57.930 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:58.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:35:58.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:35:58.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:58.061 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:58.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:35:58.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:35:58.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:58.190 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:58.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:35:58.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:35:58.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:58.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:58.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:58.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:58.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:58.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:58.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:35:58.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:35:58.322 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:35:58.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:58.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:35:58.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:59.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:59.663 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:00.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:36:00.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:00.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:00.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:00.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:00.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:00.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:00.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:36:00.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:36:00.016 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:36:00.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:00.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:36:00.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:00.391 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:00.391 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:00.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:00.392 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:00.393 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:00.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:36:00.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:36:00.394 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:36:00.394 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:00.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:36:00.737 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:00.738 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:00.738 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:01.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:01.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:01.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:01.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:01.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:01.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:01.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:01.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:01.520 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:01.520 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:01.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:01.962 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:01.962 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:01.962 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:01.963 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:36:01.963 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:36:01.963 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:36:01.963 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:36:01.964 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:36:02.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:36:02.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:36:02.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:02.311 INFO:tasks.workunit.client.0.smithi118.stdout:blackholing osds 0 2 2020-03-11T22:36:02.647 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:02.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:35:41.625033+0000 27'100 27:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:02.648 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:02.649 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:02.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:02.658 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:36:02.658 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:02.659 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:36:02.744 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:36:02.744 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:02.745 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:36:02.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:02.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:36:02.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:02.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:02.754 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:02.754 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:02.754 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:02.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:36:02.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:02.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:36:02.841 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:36:02.842 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:02.842 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:36:02.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:36:02.849 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:36:02.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:36:02.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:36:02.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:36:02.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:36:02.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:36:02.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:36:02.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stdout:writing divergent object 2020-03-11T22:36:02.855 INFO:tasks.workunit.client.0.smithi118.stdout:create missing divergent object 2020-03-11T22:36:03.202 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr:0 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:36:03.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:36:03.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:36:03.204 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:36:03.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:03.207 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:03.208 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:36:03.208 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17426/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:36:03.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:36:03.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:03.301 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:03.302 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:03.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:36:03.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:36:03.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:36:03.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:36:03.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:36:13.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:36:13.416 INFO:tasks.workunit.client.0.smithi118.stderr:rados: no process found 2020-03-11T22:36:13.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:36:13.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:13.417 INFO:tasks.workunit.client.0.smithi118.stdout:killing all the osds 2020-03-11T22:36:13.748 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:13.748 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:36:03.708631+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:13.748 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:13.749 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:13.750 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:13.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:36:13.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:13.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:13.759 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:13.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:13.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:13.865 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:13.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:36:13.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:36:14.390 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already down. 2020-03-11T22:36:14.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:36:14.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:36:14.890 INFO:tasks.workunit.client.0.smithi118.stderr:osd.1 is already down. 2020-03-11T22:36:14.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:36:14.899 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:36:15.418 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already down. 2020-03-11T22:36:15.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:15.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:36:15.919 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already out. 2020-03-11T22:36:15.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:15.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:36:16.428 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already out. 2020-03-11T22:36:16.436 INFO:tasks.workunit.client.0.smithi118.stdout:bringing up non_divergent 0 2 2020-03-11T22:36:16.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:36:16.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:16.755 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:16.755 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:36:03.708631+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:16.755 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:16.755 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:16.756 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:16.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:16.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:36:16.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:16.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:16.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:16.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:16.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:16.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:36:16.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:36:16.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:16.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:16.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:16.774 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:36:16.793 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:16.790+0000 7f6a47a05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:16.795 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:16.791+0000 7f6a47a05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:16.796 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:16.793+0000 7f6a47a05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:36:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:17.114 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:36:17.313 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:17.309+0000 7f6a47a05ec0 -1 Falling back to public interface 2020-03-11T22:36:17.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:18.202 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:18.198+0000 7f6a47a05ec0 -1 osd.0 27 log_to_monitors {default=true} 2020-03-11T22:36:18.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:18.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:18.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:18.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:18.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:18.486 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:36:18.852 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up out weight 0 up_from 33 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/22549,v1:127.0.0.1:6803/22549] [v2:127.0.0.1:6804/22549,v1:127.0.0.1:6805/22549] exists,up a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:36:18.852 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:36:18.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:18.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:18.855 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:18.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:18.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:36:18.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:18.859 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:18.859 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:18.878 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:18.874+0000 7f823a935ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:18.879 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:18.876+0000 7f823a935ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:18.880 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:18.877+0000 7f823a935ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:19.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:36:19.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:19.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:19.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:19.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:19.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:19.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:19.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:19.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:19.201 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:36:19.396 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:19.392+0000 7f823a935ec0 -1 Falling back to public interface 2020-03-11T22:36:19.567 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:20.314 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:20.311+0000 7f823a935ec0 -1 osd.2 27 log_to_monitors {default=true} 2020-03-11T22:36:20.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:20.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:20.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:20.569 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:36:20.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:20.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:20.932 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up out weight 0 up_from 34 up_thru 0 down_at 30 last_clean_interval [15,27) [v2:127.0.0.1:6810/22743,v1:127.0.0.1:6811/22743] [v2:127.0.0.1:6812/22743,v1:127.0.0.1:6813/22743] exists,up e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:36:20.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:20.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:20.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:20.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:20.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:36:21.443 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already in. 2020-03-11T22:36:21.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:21.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:36:22.002 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already in. 2020-03-11T22:36:22.011 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:22.011 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:36:22.011 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:22.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:22.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:22.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:22.012 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:22.012 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:22.013 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:22.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:22.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:22.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:22.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:22.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:22.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:22.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:22.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:22.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:22.410 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:36:22.410 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:36:22.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:22.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:22.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:22.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:36:22.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771' 2020-03-11T22:36:22.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:22.538 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:22.627 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:36:22.635 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:36:22.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1-' 2020-03-11T22:36:22.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:22.636 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:22.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:36:22.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1- 2-146028888067' 2020-03-11T22:36:22.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:22.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-141733920771 2020-03-11T22:36:22.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:22.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:22.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-141733920771 2020-03-11T22:36:22.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:22.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:36:22.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2020-03-11T22:36:22.768 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:22.768 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 141733920771 2020-03-11T22:36:23.108 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 141733920771 2020-03-11T22:36:23.108 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:24.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:24.110 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:24.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920772 -lt 141733920771 2020-03-11T22:36:24.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:24.462 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:36:24.462 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:24.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:24.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:36:24.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:36:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:36:24.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:24.465 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:36:24.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:36:24.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:36:24.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:24.848 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-146028888067 2020-03-11T22:36:24.848 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:24.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:24.849 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-146028888067 2020-03-11T22:36:24.849 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:24.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:36:24.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2020-03-11T22:36:24.850 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:24.851 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 146028888067 2020-03-11T22:36:25.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:36:25.193 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:25.194 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:25.194 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:25.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:25.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:25.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:25.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:25.640 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:25.640 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:25.640 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:25.640 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:25.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:25.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:25.971 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:25.971 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:26.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:36:26.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:26.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:26.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:26.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:26.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:26.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:26.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:26.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:26.840 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:26.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:26.841 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:26.841 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:27.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:27.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:27.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:27.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:27.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:27.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:27.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:27.282 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:27.282 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:27.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:27.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:27.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:36:27.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:27.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:36:27.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:27.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:27.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:27.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:27.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:27.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:27.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:27.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:28.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:28.252 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:28.252 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:28.253 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:28.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:28.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:28.697 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:29.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:29.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:29.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:36:29.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:29.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:36:29.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:29.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:29.535 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:29.535 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:29.535 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:29.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:29.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:29.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:29.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:29.874 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:29.875 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:29.875 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:30.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:30.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:30.313 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:30.313 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:30.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:30.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:30.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:36:30.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:30.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:36:31.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:31.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:31.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:31.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:31.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:31.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:31.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:31.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:31.889 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:31.889 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:31.889 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:31.889 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:32.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:32.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:32.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:32.327 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:32.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:32.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:32.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:36:32.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:32.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:36:34.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:34.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:34.366 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:34.366 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:34.366 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:34.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:34.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:34.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:34.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:34.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:34.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:34.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:35.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:35.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:35.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:35.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:35.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:35.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:35.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:35.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:35.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:35.587 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:35.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:35.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:36:35.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:35.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:36:38.788 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:38.788 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:38.789 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:38.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:38.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:38.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:38.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:38.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:39.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:39.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:39.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:39.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:39.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:39.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:39.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:39.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:40.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:40.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:40.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:36:40.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:40.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:36:46.406 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:46.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:46.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:46.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:46.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:46.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:46.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:46.409 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:46.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:46.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:46.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:46.747 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:47.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:47.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:47.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:47.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:47.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:47.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:47.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:47.193 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:47.193 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:47.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:47.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:47.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:36:47.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:47.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:36:54.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:54.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:54.938 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:54.938 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:54.938 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:54.939 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:54.939 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:54.939 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:55.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:55.277 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:55.278 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:55.278 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:55.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:55.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:55.744 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:55.744 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:56.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:56.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:56.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:36:56.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:36:56.624 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:36:56.624 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "cb0eae59-b0bf-4a06-b516-bd864fd3d13c", 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:36:56.620837+0000", 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:36:56.625 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:36:56.626 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:36:56.627 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:36:56.628 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:36:56.629 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.629 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:36:56.629 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.629 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:36:56.630 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.631 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:36:56.632 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.633 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.634 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:35:17.843079Z", 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:35:17.843079Z", 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.637 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:36:56.638 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:36:56.639 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:36:56.640 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 37, 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:35:18.573120+0000", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:36:21.929926+0000", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:36:20.393436+0000", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:36:21.762943+0000", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:36:56.641 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:36:56.642 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 7, 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:36:56.643 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.644 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:35:39.912315+0000", 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 1, 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool", 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:36:56.645 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 0, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 1, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 1, 2020-03-11T22:36:56.646 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 1, 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 1, 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:36:56.647 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "27", 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:36:56.648 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 0, 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 0, 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [], 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:36:56.649 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:36:56.650 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "", 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:36:56.651 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:36:56.652 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 0, 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.653 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "a11e7915-ce48-4dfd-9e2b-369bb6b4c050", 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:36:56.654 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 27, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 33, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 36, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 28, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:36:56.655 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.656 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.657 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:56.659 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.660 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:56.661 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.662 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22549 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/22549", 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/22549", 2020-03-11T22:36:56.663 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22549", 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22549", 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.664 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "d70199f6-ee80-4d1e-87ec-b3fe4e92a1de", 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:36:56.665 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 22, 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 29, 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.666 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.667 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.668 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:56.669 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.670 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.671 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.672 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 18218 2020-03-11T22:36:56.673 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/18218", 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/18218", 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18218", 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18218", 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:36:56.674 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "e5f169f3-9161-49aa-ac04-828defbd3e0d", 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:36:56.675 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 27, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 34, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:36:56.676 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.677 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.678 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.679 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:56.680 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.681 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.682 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.683 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.684 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 22743 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/22743", 2020-03-11T22:36:56.685 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/22743", 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22743", 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22743", 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:36:56.686 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:36:56.687 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:36:14.161430+0000", 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:25.486376+0000", 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:36:56.688 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:36:14.695102+0000", 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:36:56.689 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:31.595051+0000", 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:36:56.690 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:36:15.193864+0000", 2020-03-11T22:36:56.691 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:36:56.691 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:36:56.691 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:36:56.691 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:36:56.692 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:37.864514+0000", 2020-03-11T22:36:56.692 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:36:56.692 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.692 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:36:56.695 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/22549,v1:127.0.0.1:6805/22549]", 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:36:56.696 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:56.697 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:56.698 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:36:56.699 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/22549,v1:127.0.0.1:6803/22549]", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22549,v1:127.0.0.1:6809/22549]", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22549,v1:127.0.0.1:6807/22549]", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:36:56.700 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:36:56.701 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:36:56.702 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/18218,v1:127.0.0.1:6813/18218]", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:56.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:56.704 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:56.705 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/18218,v1:127.0.0.1:6811/18218]", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18218,v1:127.0.0.1:6817/18218]", 2020-03-11T22:36:56.706 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18218,v1:127.0.0.1:6815/18218]", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:56.707 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:56.708 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:36:56.708 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:36:56.709 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:56.709 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:36:56.709 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.709 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.709 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/22743,v1:127.0.0.1:6813/22743]", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:56.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:56.711 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:56.711 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:56.711 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:56.712 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:56.712 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:36:56.712 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:56.712 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:56.713 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:56.713 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:36:56.714 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/22743,v1:127.0.0.1:6811/22743]", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:36:56.715 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22743,v1:127.0.0.1:6817/22743]", 2020-03-11T22:36:56.716 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22743,v1:127.0.0.1:6815/22743]", 2020-03-11T22:36:56.716 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:36:56.716 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:36:56.716 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:56.716 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:36:56.718 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 37, 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:36:56.719 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.720 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:36:56.721 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.722 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:36:56.723 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:36:56.724 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:36:56.725 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.726 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.727 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:36:56.728 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.729 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.729 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:36:56.729 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.729 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:36:56.729 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:36:56.730 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.731 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:36:56.732 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.733 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:36:56.734 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.735 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:36:56.736 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:36:56.737 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.738 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.739 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.740 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:36:56.741 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:36:56.742 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.743 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:36:56.744 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:36:56.745 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 0, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:36:56.746 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.747 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:36:56.748 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:36:56.749 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.750 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:36:56.751 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 1, 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:36:56.752 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:36:56.753 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:36:56.754 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:56.755 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:36:56.756 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:56.757 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:56.758 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:56.758 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:36:56.758 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:36:56.759 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.759 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:36:56.759 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:36:56.759 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:36:56.759 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:36:56.760 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:36:56.760 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:36:56.760 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:36:56.760 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 16, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:56.761 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:36:56.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178496, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32512, 2020-03-11T22:36:56.763 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311394304, 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867767296, 2020-03-11T22:36:56.764 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33292288, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 388742, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15450, 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210022 2020-03-11T22:36:56.765 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 1, 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:36:56.766 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:36:56.767 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:36:56.768 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178496, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32512, 2020-03-11T22:36:56.769 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311394304, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867767296, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:56.770 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33292288, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 388742, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15450, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210022 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.771 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 1, 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:36:56.772 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.773 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:36:56.774 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:36:56.775 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:36:56.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:56.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:36:56.778 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:56.779 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:36:56.780 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19660800, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 144900, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:36:56.781 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 16, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:36:56.782 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 141733920778 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.783 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672975 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 146028888073 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.784 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 1 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.786 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.787 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 1, 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.788 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:36:56.789 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:35:18.573436+0000", 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 301, 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:36:56.790 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:36:56.791 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:36:56.791 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:36:56.792 INFO:tasks.workunit.client.0.smithi118.stderr:report 3397399028 2020-03-11T22:36:56.792 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:36:56.793 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:36:56.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:36:56.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:36:56.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:56.793 INFO:tasks.workunit.client.0.smithi118.stdout:writing non-divergent object existing_4 2020-03-11T22:36:56.955 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:56.955 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:22.043164+0000 27'100 37:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:56.955 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:56.955 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:56.956 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:56.964 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:36:57.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:36:57.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:57.046 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery 2020-03-11T22:36:57.363 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:57.363 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:22.043164+0000 27'100 37:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:57.364 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:57.364 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:57.365 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:57.373 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:57.374 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:36:57.374 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:57.374 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:57.374 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:57.374 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:36:57.466 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:36:57.472 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:57.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:36:57.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:57.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:57.474 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:57.474 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.475 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:57.475 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:36:57.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:57.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:36:57.564 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:36:57.571 INFO:tasks.workunit.client.0.smithi118.stdout:revive divergent 1 2020-03-11T22:36:57.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:36:57.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:57.886 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:57.887 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:22.043164+0000 27'100 37:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:36:57.887 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:36:57.887 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:57.888 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:36:57.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:36:58.539 INFO:tasks.workunit.client.0.smithi118.stderr:noup is set 2020-03-11T22:36:58.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:36:58.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:58.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:58.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:36:58.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:58.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:58.553 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:58.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:58.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:36:58.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:36:58.557 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:36:58.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:58.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:58.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:58.576 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:58.573+0000 7fae2ecafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.578 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:58.575+0000 7fae2ecafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.580 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:58.576+0000 7fae2ecafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.911 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:36:58.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:36:59.097 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:59.094+0000 7fae2ecafec0 -1 Falling back to public interface 2020-03-11T22:36:59.964 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:36:59.960+0000 7fae2ecafec0 -1 osd.1 27 log_to_monitors {default=true} 2020-03-11T22:37:03.913 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery divergent 2020-03-11T22:37:03.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:37:03.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:04.242 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:04.242 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:22.043164+0000 37'101 37:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:04.242 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:04.242 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:04.243 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:04.252 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:37:04.252 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:37:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:37:04.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:37:04.343 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:37:04.349 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:37:05.036 INFO:tasks.workunit.client.0.smithi118.stderr:noup is unset 2020-03-11T22:37:05.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:37:05.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:05.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:05.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:05.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:05.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:05.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:05.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:05.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:05.046 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:05.413 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 33 up_thru 36 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/22549,v1:127.0.0.1:6803/22549] [v2:127.0.0.1:6804/22549,v1:127.0.0.1:6805/22549] exists,up a11e7915-ce48-4dfd-9e2b-369bb6b4c050 2020-03-11T22:37:05.413 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:05.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:05.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:05.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:05.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:05.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/24211,v1:127.0.0.1:6819/24211] [v2:127.0.0.1:6820/24211,v1:127.0.0.1:6821/24211] exists,up d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:05.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:05.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:05.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:06.112 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [15,27) [v2:127.0.0.1:6810/22743,v1:127.0.0.1:6811/22743] [v2:127.0.0.1:6812/22743,v1:127.0.0.1:6813/22743] exists,up e5f169f3-9161-49aa-ac04-828defbd3e0d 2020-03-11T22:37:06.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:06.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:06.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:06.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:06.433 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:06.433 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:37:04.956890+0000 27'101 41:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:06.433 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:06.433 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:06.435 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:06.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:37:06.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:06.443 INFO:tasks.workunit.client.0.smithi118.stdout:wait for peering 2020-03-11T22:37:06.761 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:06.762 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:37:04.956890+0000 27'101 41:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:06.762 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:06.762 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:06.763 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:06.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:37:18.071 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:37:18.071 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:18.071 INFO:tasks.workunit.client.0.smithi118.stdout:killing divergent 1 2020-03-11T22:37:18.400 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:18.400 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:37:05.426584+0000 37'101 43:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:18.400 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:18.400 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:18.402 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:18.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:37:18.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:18.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:18.411 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:18.411 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:18.411 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:18.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:18.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:37:18.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:18.516 INFO:tasks.workunit.client.0.smithi118.stdout:reviving divergent 1 2020-03-11T22:37:18.833 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:18.834 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:37:05.426584+0000 37'101 43:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:18.834 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:18.834 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:18.835 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:18.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:37:18.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:18.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:18.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:18.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:18.848 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:18.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:18.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:37:18.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:18.851 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:37:18.851 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:37:18.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:37:18.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:18.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:18.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:18.872 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:18.869+0000 7f28c8065ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:18.873 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:18.871+0000 7f28c8065ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:18.874 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:18.872+0000 7f28c8065ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:19.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:37:19.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:19.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:19.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:19.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:19.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:19.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:19.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:19.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:19.207 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:19.402 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:19.399+0000 7f28c8065ec0 -1 Falling back to public interface 2020-03-11T22:37:19.551 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/24211,v1:127.0.0.1:6819/24211] [v2:127.0.0.1:6820/24211,v1:127.0.0.1:6821/24211] exists,up d70199f6-ee80-4d1e-87ec-b3fe4e92a1de 2020-03-11T22:37:19.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:19.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:19.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:19.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:37:20.319 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:20.316+0000 7f28c8065ec0 -1 osd.1 43 log_to_monitors {default=true} 2020-03-11T22:37:39.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:37:39.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:39.553 INFO:tasks.workunit.client.0.smithi118.stdout:allowing recovery 2020-03-11T22:37:39.885 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:39.885 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:37:21.727737+0000 43'102 46:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:39.885 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:39.886 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:39.887 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:39.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:39.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:37:40.015 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:40.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:40.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:37:40.143 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:40.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:40.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:37:40.271 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:40.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:37:40.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:40.279 INFO:tasks.workunit.client.0.smithi118.stdout:reading divergent objects 2020-03-11T22:37:40.597 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:40.598 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:37:21.727737+0000 43'102 46:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:39.962466+0000 0'0 2020-03-11T22:35:39.962466+0000 0 2020-03-11T22:37:40.598 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:37:40.598 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:40.599 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:37:40.608 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:37:40.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:37:40.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:40.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:37:40.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:40.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:37:40.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:40.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:37:40.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:40.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:37:40.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:37:40.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:37:40.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:37:40.736 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:36:02.874174+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:40.737 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:36:02.874174+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:40.737 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:37:40.737 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'101 2020-03-11T22:37:40.738 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:37:40.738 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:37:40.738 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 27'101 2020-03-11T22:37:40.738 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:36:02.874174+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:40.738 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:05.360+0000 7fae0ce6a700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:37:40.739 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:37:40.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi118.stdout:success 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:37:41.203 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' does not exist 2020-03-11T22:37:41.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:37:41.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:41.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:41.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:41.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:41.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:41.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:41.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:37:41.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:37:41.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:37:41.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:37:41.320 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:41.320 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:41.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:41.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:41.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:41.323 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:41.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:37:41.324 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:37:41.324 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:37:41.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:37:41.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:37:41.326 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:37:41.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:41.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:37:41.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:37:41.328 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:41.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:37:41.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:37:41.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:37:41.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:37:41.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:37:41.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:41.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:41.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:41.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:41.345 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:37:41.345 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:37:41.345 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:37:41.345 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:37:41.345 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:37:41.354 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:37:41.354 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:41.354 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:37:41.354 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:37:41.354 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17426 2020-03-11T22:37:41.357 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:37:41.357 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:37:41.363 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:37:41.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:37:41.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:37:41.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:37:41.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:37:41.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:37:41.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:37:41.365 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:37:41.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:37:41.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:37:41.367 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:37:41.367 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:37:41.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:37:41.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:37:41.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:37:41.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:37:41.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:37:41.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:37:41.372 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:37:41.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:37:41.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:41.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:41.667 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:41.667 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.667 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:41.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:37:41.837 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:37:41.838 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:37:41.838 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:41.839 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:41.840 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.840 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.840 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:37:41.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:41.841 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get fsid 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:37:41.926 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:41.927 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:37:41.928 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:41.928 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get mon_host 2020-03-11T22:37:42.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:37:42.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:37:42.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:37:42.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:37:42.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:37:42.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:37:42.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:37:42.886 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:42.887 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:42.888 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:42.888 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:42.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:42.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:37:42.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:42.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:42.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:42.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:42.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:42.920 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:42.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:37:42.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 847b0c18-b574-4952-9cbd-7148f90e9b63' 2020-03-11T22:37:42.921 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:42.922 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:37:42.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA2aGleceOKNxAAZpVWZ8nwppI/qtyoyDEdaw== 2020-03-11T22:37:42.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA2aGleceOKNxAAZpVWZ8nwppI/qtyoyDEdaw=="}' 2020-03-11T22:37:42.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 847b0c18-b574-4952-9cbd-7148f90e9b63 -i td/divergent-priors/0/new.json 2020-03-11T22:37:43.309 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:43.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:37:43.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA2aGleceOKNxAAZpVWZ8nwppI/qtyoyDEdaw== --osd-uuid 847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:37:43.348 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.344+0000 7f703ca98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:43.350 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.346+0000 7f703ca98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:43.351 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.347+0000 7f703ca98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:43.352 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.348+0000 7f703ca98ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:43.352 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.348+0000 7f703ca98ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:43.352 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.348+0000 7f703ca98ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:43.395 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:43.391+0000 7f703ca98ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:37:46.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:37:46.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:46.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:37:46.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:46.331 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:37:46.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:37:46.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:46.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:46.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:46.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:46.776 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:37:46.803 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:46.799+0000 7f7be7dedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:46.801+0000 7f7be7dedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:46.803+0000 7f7be7dedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:47.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:37:47.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:47.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:47.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:47.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:47.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:47.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:47.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:47.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:47.123 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:47.322 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:47.318+0000 7f7be7dedec0 -1 Falling back to public interface 2020-03-11T22:37:47.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:48.080 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:48.076+0000 7f7be7dedec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:37:48.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:48.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:48.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:48.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:48.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:48.493 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:37:48.879 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25513,v1:127.0.0.1:6803/25513] [v2:127.0.0.1:6804/25513,v1:127.0.0.1:6805/25513] exists,up 847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:48.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:48.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:48.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:48.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:48.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:37:48.885 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 aa764004-f9e8-4507-a028-b30c8088b6f2' 2020-03-11T22:37:48.886 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:48.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA8aGles8dVNRAAYrg90X+hNrFuJhvq5OZJZw== 2020-03-11T22:37:48.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA8aGles8dVNRAAYrg90X+hNrFuJhvq5OZJZw=="}' 2020-03-11T22:37:48.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new aa764004-f9e8-4507-a028-b30c8088b6f2 -i td/divergent-priors/1/new.json 2020-03-11T22:37:49.444 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:37:49.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:37:49.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8aGles8dVNRAAYrg90X+hNrFuJhvq5OZJZw== --osd-uuid aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:37:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.478+0000 7fbd57744ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:49.483 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.480+0000 7fbd57744ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:49.485 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.481+0000 7fbd57744ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:49.486 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.482+0000 7fbd57744ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:49.486 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.482+0000 7fbd57744ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:49.486 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.482+0000 7fbd57744ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:49.587 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:49.584+0000 7fbd57744ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:37:52.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:37:52.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:52.687 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:37:52.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:37:52.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:53.156 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:37:53.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:37:53.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:53.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:53.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:53.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:53.183 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:53.179+0000 7f55099e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.184 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:53.181+0000 7f55099e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.186 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:53.183+0000 7f55099e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.500 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:37:53.500 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:53.500 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:53.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:53.502 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:37:53.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:53.698+0000 7f55099e0ec0 -1 Falling back to public interface 2020-03-11T22:37:53.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:54.437 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:54.434+0000 7f55099e0ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:37:54.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:54.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:54.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:54.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:54.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:54.851 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:37:55.205 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25868,v1:127.0.0.1:6811/25868] [v2:127.0.0.1:6812/25868,v1:127.0.0.1:6813/25868] exists,up aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:37:55.205 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:37:55.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:55.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:55.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:55.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:55.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:37:55.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:55.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:55.208 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:55.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:55.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:37:55.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d0bbeb21-437a-424d-9469-615d7849a1d4' 2020-03-11T22:37:55.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:55.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBDaGleLKsqDRAA/Hr5f2Y++95btzR9eAnPsQ== 2020-03-11T22:37:55.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBDaGleLKsqDRAA/Hr5f2Y++95btzR9eAnPsQ=="}' 2020-03-11T22:37:55.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d0bbeb21-437a-424d-9469-615d7849a1d4 -i td/divergent-priors/2/new.json 2020-03-11T22:37:55.783 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:37:55.792 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:37:55.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBDaGleLKsqDRAA/Hr5f2Y++95btzR9eAnPsQ== --osd-uuid d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:37:55.822 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.818+0000 7f2cd74b5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:55.823 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.819+0000 7f2cd74b5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:55.825 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.821+0000 7f2cd74b5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:55.826 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.821+0000 7f2cd74b5ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:55.826 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.821+0000 7f2cd74b5ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:55.826 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.821+0000 7f2cd74b5ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:55.926 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:55.923+0000 7f2cd74b5ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:37:58.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:37:58.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:58.999 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:37:58.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:37:58.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:59.456 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:37:59.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:37:59.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:59.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:59.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:59.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:59.483 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:59.480+0000 7f3276d00ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.485 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:59.482+0000 7f3276d00ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.486 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:59.483+0000 7f3276d00ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:37:59.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:59.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:59.800 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:38:00.001 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:37:59.999+0000 7f3276d00ec0 -1 Falling back to public interface 2020-03-11T22:38:00.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:00.744 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:00.741+0000 7f3276d00ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:38:01.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:01.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:01.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:01.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:01.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:01.147 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:38:01.518 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26223,v1:127.0.0.1:6819/26223] [v2:127.0.0.1:6820/26223,v1:127.0.0.1:6821/26223] exists,up d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:38:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:38:02.280 INFO:tasks.workunit.client.0.smithi118.stderr:noout is set 2020-03-11T22:38:02.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:38:02.914 INFO:tasks.workunit.client.0.smithi118.stderr:noin is set 2020-03-11T22:38:02.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:38:03.549 INFO:tasks.workunit.client.0.smithi118.stderr:nodown is set 2020-03-11T22:38:03.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:38:03.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:38:04.108 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' already exists 2020-03-11T22:38:04.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:38:05.116 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:38:05.746 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 3 2020-03-11T22:38:05.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:38:06.614 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 min_size to 2 2020-03-11T22:38:06.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:38:06.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:06.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:06.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:06.972 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:38:06.972 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:38:06.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:06.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:06.973 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:07.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:38:07.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:38:07.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:07.105 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:07.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:38:07.233 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:38:07.233 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:07.233 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:07.360 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:38:07.360 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:38:07.361 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:07.361 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:38:07.362 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:07.362 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:07.362 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:38:07.362 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:07.363 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:38:07.363 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:38:07.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:07.364 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:38:07.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:38:07.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:08.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:09.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:38:09.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:09.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:09.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:09.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:09.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:09.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:38:09.396 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:38:09.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:09.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:09.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:09.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:09.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:09.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:09.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:38:09.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:38:09.400 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:09.400 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:38:09.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:38:09.761 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:09.762 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:09.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:09.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:09.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:09.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:09.852 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:10.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:10.191 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:38:10.191 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:38:10.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:10.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:10.192 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:10.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:38:10.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:38:10.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:10.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:10.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:38:10.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:38:10.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:10.450 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:10.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:38:10.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-03-11T22:38:10.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:10.579 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:38:10.579 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:10.580 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:10.580 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:38:10.580 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:10.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:38:10.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:38:10.581 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:10.582 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:38:10.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:38:10.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:11.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:11.934 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:12.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:38:12.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:12.286 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:38:12.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:12.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:12.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:38:12.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:12.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:38:12.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:38:12.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:12.289 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:38:12.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-11T22:38:12.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:12.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:12.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:12.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:12.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:12.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:12.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:38:12.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:38:12.647 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:12.647 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:38:13.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:38:13.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:13.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:13.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:13.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:13.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:13.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:13.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:13.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:13.481 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:13.481 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:13.481 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:13.810 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:13.811 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:13.811 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:13.811 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:14.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:14.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:14.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:14.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:38:14.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:38:14.585 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:38:14.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:38:14.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:38:14.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:14.594 INFO:tasks.workunit.client.0.smithi118.stdout:primary and soon to be divergent is 1 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:14.913 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:38:04.860577+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:14.914 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:38:14.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:38:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:38:14.924 INFO:tasks.workunit.client.0.smithi118.stdout:writing initial objects 2020-03-11T22:38:14.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:14.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:38:15.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:38:15.054 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.054 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:38:15.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:38:15.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:38:15.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:38:15.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:38:15.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:38:15.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:38:15.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:38:15.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:38:15.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:38:15.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:38:15.712 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.712 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:38:15.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.754 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:38:15.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:38:15.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:38:15.962 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:15.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:38:16.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:38:16.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:38:16.087 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.088 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:38:16.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:38:16.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:38:16.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:38:16.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:38:16.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:38:16.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:38:16.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:38:16.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:38:16.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:38:16.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:16.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:38:17.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:38:17.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:38:17.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:38:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:38:17.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:38:17.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:38:17.388 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.388 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:38:17.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:38:17.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:38:17.579 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.580 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:38:17.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:38:17.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:38:17.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:38:17.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:38:17.788 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.788 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:38:17.899 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.900 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:38:17.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:17.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:38:18.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:38:18.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:38:18.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:38:18.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:38:18.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:38:18.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:38:18.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:38:18.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:38:18.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:38:18.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:38:18.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:38:18.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:38:18.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:38:18.971 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:18.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:38:19.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:38:19.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:38:19.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:38:19.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:38:19.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:38:19.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:38:19.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:38:19.419 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.419 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:38:19.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:38:19.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:38:19.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:38:19.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:38:19.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:38:19.788 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.789 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:38:19.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:38:19.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:38:19.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:38:19.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:19.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:38:20.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:38:20.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:38:20.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:38:20.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:38:20.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:38:20.404 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.405 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:38:20.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:38:20.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:38:20.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:38:20.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:38:20.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:38:20.664 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.664 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:38:20.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:38:20.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:38:20.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:38:20.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:38:20.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:20.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:38:21.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:21.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:38:21.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:21.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:21.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:21.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:21.124 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:21.124 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:21.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:21.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:21.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:21.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:21.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:21.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:21.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:21.180 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:21.573 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:21.573 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:38:21.573 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:38:21.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:21.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:21.574 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:21.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:38:21.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:38:21.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:21.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:21.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:38:21.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:38:21.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:21.830 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:21.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:38:21.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450' 2020-03-11T22:38:21.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:21.959 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:38:21.959 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:21.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:21.960 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:38:21.961 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:21.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:38:21.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:38:21.962 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:38:21.962 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:22.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:38:22.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:22.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:22.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:22.323 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:22.324 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:22.324 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:22.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:38:22.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:38:22.325 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:22.326 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:38:22.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:38:22.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:22.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:38:22.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:22.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:22.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:38:22.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:22.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:38:22.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:38:22.668 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:22.668 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:38:23.033 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509450 2020-03-11T22:38:23.034 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:23.034 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:23.034 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:23.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:23.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:23.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:23.825 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:23.825 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:23.826 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:24.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:24.396 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:24.396 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:24.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:38:24.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:38:24.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:38:24.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:38:24.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:38:24.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:38:24.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:38:24.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:24.625 INFO:tasks.workunit.client.0.smithi118.stdout:blackholing osds 0 2 2020-03-11T22:38:24.945 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.945 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:38:04.860577+0000 28'100 28:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:38:24.945 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:38:24.945 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.946 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:38:24.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:24.954 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:38:24.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:24.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:38:25.041 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:38:25.041 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:25.041 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:38:25.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:25.050 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:38:25.051 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:38:25.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:25.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:38:25.135 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:38:25.136 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:25.136 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:38:25.142 INFO:tasks.workunit.client.0.smithi118.stdout:create new divergent objects 2020-03-11T22:38:25.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:38:25.142 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:38:25.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:38:25.144 INFO:tasks.workunit.client.0.smithi118.stdout:writing divergent objects 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:38:25.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:25.147 INFO:tasks.workunit.client.0.smithi118.stdout:remove divergent objects 2020-03-11T22:38:25.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:38:25.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:38:25.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:25.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:38:25.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:38:25.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:38:25.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:38:25.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:38:35.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:38:35.153 INFO:tasks.workunit.client.0.smithi118.stderr:rados: no process found 2020-03-11T22:38:35.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:38:35.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:35.154 INFO:tasks.workunit.client.0.smithi118.stdout:killing all the osds 2020-03-11T22:38:35.477 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:35.478 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:38:04.860577+0000 28'106 28:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:38:35.478 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:38:35.478 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:38:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:38:35.488 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:35.488 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:35.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:35.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:35.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:35.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:38:36.088 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already down. 2020-03-11T22:38:36.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:36.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:38:36.552 INFO:tasks.workunit.client.0.smithi118.stderr:osd.1 is already down. 2020-03-11T22:38:36.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:36.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:38:37.185 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already down. 2020-03-11T22:38:37.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:37.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:38:37.703 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already out. 2020-03-11T22:38:37.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:37.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:38:38.195 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already out. 2020-03-11T22:38:38.203 INFO:tasks.workunit.client.0.smithi118.stdout:bringing up non_divergent 0 2 2020-03-11T22:38:38.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:38:38.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:38.525 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:38.525 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:38:04.860577+0000 28'106 28:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:38:38.525 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:38:38.525 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:38.526 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:38:38.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:38.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:38:38.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:38.540 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:38.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:38.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:38.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:38.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:38.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:38.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:38.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:38:38.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:38:38.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:38.543 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:38:38.543 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:38:38.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:38:38.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:38.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:38.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:38.564 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:38.560+0000 7f5a2e70fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:38.565 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:38.562+0000 7f5a2e70fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:38.567 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:38.564+0000 7f5a2e70fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:38.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:38:38.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:38.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:38.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:38.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:38.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:38.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:38.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:38.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:38.883 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:38:39.083 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:39.079+0000 7f5a2e70fec0 -1 Falling back to public interface 2020-03-11T22:38:39.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:39.971 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:39.968+0000 7f5a2e70fec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:38:40.227 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:40.227 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:40.227 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:40.227 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:40.227 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:40.228 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:38:40.583 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30238,v1:127.0.0.1:6803/30238] [v2:127.0.0.1:6804/30238,v1:127.0.0.1:6805/30238] exists,up 847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:38:40.583 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:38:40.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:40.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:40.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:40.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:40.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:40.608 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:40.606+0000 7f95732e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:40.610 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:40.607+0000 7f95732e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:40.611 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:40.608+0000 7f95732e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:40.925 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:38:40.925 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:40.925 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:40.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:40.927 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:38:41.127 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:41.125+0000 7f95732e0ec0 -1 Falling back to public interface 2020-03-11T22:38:41.301 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:42.021 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:38:42.018+0000 7f95732e0ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:38:42.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:42.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:42.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:42.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:42.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:42.303 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:38:42.654 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30432,v1:127.0.0.1:6811/30432] [v2:127.0.0.1:6812/30432,v1:127.0.0.1:6813/30432] exists,up d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:38:42.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:42.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:42.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:42.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:42.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:38:43.194 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already in. 2020-03-11T22:38:43.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:43.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:38:43.770 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already in. 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:43.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:43.780 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:43.780 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:43.781 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:43.781 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:43.781 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:43.781 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:43.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:43.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:43.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:43.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:43.836 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:44.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:44.181 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:38:44.181 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:38:44.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:44.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:44.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:44.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:38:44.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:38:44.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:44.311 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:44.402 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:38:44.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:38:44.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:38:44.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:44.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:44.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:38:44.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:38:44.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:44.540 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:44.540 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:44.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:44.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:44.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:44.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:38:44.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:38:44.542 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:44.543 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:38:44.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:38:44.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:45.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:45.884 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:46.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:38:46.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:47.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:38:47.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:47.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:38:47.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:47.593 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:38:47.593 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:47.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:47.594 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:38:47.594 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:47.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:38:47.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:38:47.595 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:47.595 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:38:47.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:38:47.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:38:47.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:47.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:47.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:47.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:47.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:47.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:47.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:38:47.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:38:47.944 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:47.945 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:38:48.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:38:48.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:48.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:48.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:48.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:48.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:48.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:49.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:49.063 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:49.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:49.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:49.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:49.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:38:49.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:38:49.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:38:49.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:49.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:38:49.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:49.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:49.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:49.937 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:49.938 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:49.938 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:50.383 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:50.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:50.385 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:50.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:50.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:50.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:38:50.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:50.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:38:51.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:51.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:51.357 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:51.358 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:51.358 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:51.358 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:51.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:51.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:51.800 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:51.800 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:52.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:52.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:52.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:38:52.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:52.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:38:52.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:52.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:52.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:52.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:52.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:52.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:52.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:52.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:52.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:52.975 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:52.975 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:52.975 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:53.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:53.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:53.417 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:53.417 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:53.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:38:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:53.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:38:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:54.658 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:54.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:54.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:54.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:54.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:54.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:54.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:54.994 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:54.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:54.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:55.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:55.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:55.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:55.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:55.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:55.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:55.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:55.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:55.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:55.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:38:55.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:55.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:38:57.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:57.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:57.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:57.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:57.836 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:57.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:57.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:58.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:58.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:58.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:58.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:58.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:58.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:58.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:38:58.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:58.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:39:01.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:01.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:01.941 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:01.941 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:01.941 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:01.941 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:01.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:01.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:02.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:02.280 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:02.280 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:02.280 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:02.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:02.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:02.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:02.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:02.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:02.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:02.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:02.736 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:02.736 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:03.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:03.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:03.185 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:39:03.185 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:03.185 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:39:09.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:09.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:09.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:09.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:09.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:09.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:09.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:09.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:09.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:09.921 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:09.922 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:09.922 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:10.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:10.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:10.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:10.380 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:10.823 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:10.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:10.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:39:10.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:10.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:18.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:18.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:18.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:18.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:18.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:18.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:18.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:18.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:18.472 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:18.472 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:18.473 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:18.473 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:18.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:18.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:18.922 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:18.923 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:19.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:19.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:19.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:19.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:19.808 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "699bfc45-3c83-4fa8-a3b2-193b7fa728f8", 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:39:19.805592+0000", 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:39:19.809 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:19.810 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:19.811 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.812 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:39:19.813 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.814 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.815 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.816 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:19.817 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:37:41.375086Z", 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:37:41.375086Z", 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:39:19.818 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:39:19.819 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:39:19.820 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.821 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.822 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 38, 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:39:19.823 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:37:42.115550+0000", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:38:43.698720+0000", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:38:42.096662+0000", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:38:43.508670+0000", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:39:19.824 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.825 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 7, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:39:19.826 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:39:19.827 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:38:03.813323+0000", 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 1, 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 0, 2020-03-11T22:39:19.828 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 1, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 1, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 1, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 1, 2020-03-11T22:39:19.829 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:39:19.830 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "28", 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:19.831 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 0, 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 0, 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [], 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:39:19.832 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:19.833 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "", 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:39:19.834 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 0, 2020-03-11T22:39:19.835 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:39:19.836 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "847b0c18-b574-4952-9cbd-7148f90e9b63", 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:19.837 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 34, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 37, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 29, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.838 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.839 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.840 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:19.841 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.842 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.843 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.844 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:19.845 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/30238", 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/30238", 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30238", 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30238", 2020-03-11T22:39:19.846 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "aa764004-f9e8-4507-a028-b30c8088b6f2", 2020-03-11T22:39:19.847 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:39:19.848 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 22, 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.849 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.850 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:19.851 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.852 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:19.853 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.854 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.855 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.856 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/25868", 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/25868", 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25868", 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25868", 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.857 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "d0bbeb21-437a-424d-9469-615d7849a1d4", 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:19.858 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 35, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:19.859 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.860 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:19.861 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.862 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:19.863 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.864 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:19.865 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.866 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.867 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/30432", 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/30432", 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30432", 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30432", 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.868 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:35.894389+0000", 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:19.869 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:49.081495+0000", 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.870 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:36.367679+0000", 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:55.431828+0000", 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:36.936826+0000", 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:01.714786+0000", 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:39:19.873 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:39:19.874 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:39:19.875 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:19.876 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/30238,v1:127.0.0.1:6805/30238]", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:19.877 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:19.878 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:19.879 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:19.880 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/30238,v1:127.0.0.1:6803/30238]", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30238,v1:127.0.0.1:6809/30238]", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30238,v1:127.0.0.1:6807/30238]", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:19.881 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:19.882 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/25868,v1:127.0.0.1:6813/25868]", 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:19.883 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:19.884 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:19.885 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:19.886 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/25868,v1:127.0.0.1:6811/25868]", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25868,v1:127.0.0.1:6817/25868]", 2020-03-11T22:39:19.887 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25868,v1:127.0.0.1:6815/25868]", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:19.888 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.889 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/30432,v1:127.0.0.1:6813/30432]", 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:19.890 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:19.891 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:19.892 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:19.893 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/30432,v1:127.0.0.1:6811/30432]", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30432,v1:127.0.0.1:6817/30432]", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30432,v1:127.0.0.1:6815/30432]", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:19.894 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:19.895 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 38, 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:39:19.896 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.897 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:39:19.898 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:39:19.899 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:39:19.900 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.901 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.902 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.903 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:39:19.904 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:19.905 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:19.906 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:39:19.907 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:39:19.908 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:39:19.909 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:39:19.910 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:19.911 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.912 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.913 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:39:19.914 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.915 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.916 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.917 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:39:19.918 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:39:19.919 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:39:19.920 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:19.921 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:39:19.922 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 0, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:19.923 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:39:19.924 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:19.925 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.926 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 1, 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:19.930 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:19.931 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:19.932 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:19.933 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:39:19.934 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:19.935 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 16, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:39:19.936 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:39:19.937 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178816, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32768, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:39:19.938 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393984, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867439616, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33554432, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 396154, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:19.939 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15388, 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210084 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:19.940 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:39:19.941 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:39:19.942 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:39:19.943 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178816, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32768, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393984, 2020-03-11T22:39:19.944 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867439616, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33554432, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 396154, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:19.945 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15388, 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210084 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:39:19.946 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.947 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:39:19.948 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:39:19.949 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:39:19.950 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:39:19.951 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:19.952 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:39:19.953 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:19.954 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:39:19.955 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19529728, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 143228, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:39:19.956 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 16, 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 146028888074 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672974 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 150323855370 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 1 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.960 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.961 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:19.962 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 1, 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:39:19.963 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:37:42.115935+0000", 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:39:19.964 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 303, 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:19.965 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:39:19.966 INFO:tasks.workunit.client.0.smithi118.stderr:report 3639910197 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:19.967 INFO:tasks.workunit.client.0.smithi118.stdout:writing non-divergent object existing_4 2020-03-11T22:39:20.145 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:20.146 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:43.800351+0000 28'100 38:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:39:20.146 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:39:20.146 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:20.147 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:39:20.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:39:20.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:20.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:39:20.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:20.282 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:20.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:20.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:20.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:20.337 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:20.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:20.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:20.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:20.338 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:20.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:20.679 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:39:20.679 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:39:20.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:20.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:20.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:20.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888077 2020-03-11T22:39:20.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077' 2020-03-11T22:39:20.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:20.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:20.898 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:39:20.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:39:20.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1-' 2020-03-11T22:39:20.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:20.907 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:21.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:39:21.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1- 2-150323855372' 2020-03-11T22:39:21.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:21.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888077 2020-03-11T22:39:21.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:21.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:21.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888077 2020-03-11T22:39:21.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:21.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888077 2020-03-11T22:39:21.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888077' 2020-03-11T22:39:21.038 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:21.039 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 146028888077 2020-03-11T22:39:21.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888075 -lt 146028888077 2020-03-11T22:39:21.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:22.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:22.380 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:22.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888077 2020-03-11T22:39:22.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:22.743 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:39:22.743 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:22.744 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:22.744 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:39:22.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:22.745 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:39:22.745 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:39:23.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:39:23.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:39:23.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:23.090 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:39:23.090 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:23.091 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:23.091 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:39:23.091 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:23.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:39:23.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:39:23.092 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:23.092 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:39:23.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855373 -lt 150323855372 2020-03-11T22:39:23.490 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:23.491 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:23.491 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:23.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:23.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:23.931 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:23.931 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:23.931 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:23.932 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:23.932 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:23.932 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:24.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:24.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:24.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:24.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:24.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:24.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:39:24.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:39:24.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:39:24.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:24.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:39:24.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:24.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:24.806 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:24.806 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:24.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:24.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:24.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:24.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:25.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:25.139 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:25.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:25.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:25.580 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:25.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:25.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:25.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:25.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:25.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:25.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:25.582 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:25.582 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:26.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:26.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:26.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:39:26.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:26.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:39:26.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:26.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:26.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:26.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:26.548 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:26.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:26.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:26.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:26.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:26.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:26.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:26.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:26.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:26.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:26.994 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:26.994 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:27.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:27.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:27.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:39:27.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:27.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:39:27.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:27.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:27.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:27.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:27.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:27.837 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:27.838 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:27.838 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:28.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:28.181 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:28.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:28.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:28.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:28.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:28.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:28.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:28.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:28.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:28.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:28.626 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:28.627 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:29.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:29.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:29.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:39:29.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:29.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:39:29.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:29.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:29.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:29.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:29.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:29.865 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:29.865 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:29.865 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:30.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:30.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:30.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:30.201 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:31.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:31.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:31.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:39:31.087 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:31.087 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:39:32.688 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:32.688 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:32.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:32.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:32.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:32.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:32.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:32.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:33.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:33.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:33.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:33.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:33.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:33.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:33.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:33.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:33.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:33.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:39:33.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:33.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:39:37.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:37.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:37.108 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:37.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:37.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:37.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:37.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:37.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:37.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:37.449 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:37.450 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:37.450 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:37.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:37.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:37.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:37.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:37.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:37.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:37.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:37.897 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:37.897 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:38.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:38.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:38.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:39:38.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:38.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:39:44.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:44.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:44.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:45.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:45.079 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:45.079 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:45.079 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:45.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:45.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:45.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:45.522 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:45.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:45.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:45.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:39:45.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:45.960 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:53.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:53.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:53.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:53.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:53.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:53.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:54.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:54.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:54.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:54.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:54.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:54.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:54.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:54.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:54.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:54.944 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:39:54.944 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "699bfc45-3c83-4fa8-a3b2-193b7fa728f8", 2020-03-11T22:39:54.944 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:54.944 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:39:54.941049+0000", 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:54.945 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:54.946 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:54.947 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:39:54.948 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 2 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.949 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:39:54.950 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:54.951 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:39:54.951 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.951 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:39:54.952 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.953 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:54.954 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:37:41.375086Z", 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:37:41.375086Z", 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:39:54.955 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:39:54.956 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:39:54.957 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.958 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:39:54.959 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.960 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 38, 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:37:42.115550+0000", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:38:43.698720+0000", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:38:42.096662+0000", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:38:43.508670+0000", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:54.961 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 7, 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.964 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:38:03.813323+0000", 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 1, 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:39:54.965 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 0, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 1, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 1, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 1, 2020-03-11T22:39:54.966 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 1, 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:39:54.967 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "28", 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:54.968 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 0, 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 0, 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [], 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:39:54.969 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:54.970 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "", 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.971 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:54.972 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 0, 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.973 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "847b0c18-b574-4952-9cbd-7148f90e9b63", 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:54.974 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 34, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 37, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 29, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:54.975 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.976 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:54.977 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.980 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.981 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.982 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.982 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:54.982 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.982 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.982 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30238 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/30238", 2020-03-11T22:39:54.983 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/30238", 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30238", 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30238", 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.984 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "aa764004-f9e8-4507-a028-b30c8088b6f2", 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:54.985 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 22, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:54.986 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.987 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.988 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.989 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.990 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:54.991 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.992 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 25868 2020-03-11T22:39:54.993 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/25868", 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/25868", 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25868", 2020-03-11T22:39:54.994 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25868", 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:54.995 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "d0bbeb21-437a-424d-9469-615d7849a1d4", 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:39:54.996 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 35, 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.997 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:54.998 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:54.999 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.000 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.001 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:55.002 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:39:55.003 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 30432 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.004 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/30432", 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/30432", 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30432", 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30432", 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:39:55.005 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:55.006 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:35.894389+0000", 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:49.081495+0000", 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:55.007 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:36.367679+0000", 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:55.008 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:55.431828+0000", 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:38:36.936826+0000", 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:39:55.009 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:01.714786+0000", 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:39:55.010 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:39:55.011 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:55.012 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.013 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/30238,v1:127.0.0.1:6805/30238]", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:55.014 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:55.015 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:55.016 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/30238,v1:127.0.0.1:6803/30238]", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:55.017 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30238,v1:127.0.0.1:6809/30238]", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30238,v1:127.0.0.1:6807/30238]", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:55.018 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:55.019 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/25868,v1:127.0.0.1:6813/25868]", 2020-03-11T22:39:55.020 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:55.021 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:55.022 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:55.023 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/25868,v1:127.0.0.1:6811/25868]", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25868,v1:127.0.0.1:6817/25868]", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25868,v1:127.0.0.1:6815/25868]", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:55.024 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:55.025 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:39:55.026 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/30432,v1:127.0.0.1:6813/30432]", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:55.027 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/30432,v1:127.0.0.1:6811/30432]", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30432,v1:127.0.0.1:6817/30432]", 2020-03-11T22:39:55.030 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30432,v1:127.0.0.1:6815/30432]", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:55.031 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.032 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 38, 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:39:55.033 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.034 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:39:55.035 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.036 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.037 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.038 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:39:55.039 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:39:55.040 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:55.041 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:39:55.042 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:39:55.043 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:39:55.044 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:55.045 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.046 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:55.047 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:55.048 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:39:55.049 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:39:55.050 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.051 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.052 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.053 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:39:55.054 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.055 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.056 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:39:55.057 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:39:55.058 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 0, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:55.059 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:39:55.060 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:55.061 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.062 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:39:55.063 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:39:55.064 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 1, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 47947, 2020-03-11T22:39:55.065 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:55.066 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:55.067 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:55.068 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:55.069 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:55.070 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:39:55.071 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:39:55.072 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 17, 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 17, 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:39:55.073 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:55.074 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178816, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32768, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393984, 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:39:55.075 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867439616, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33554432, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 395448, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:55.076 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15388, 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210084 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:55.077 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:55.080 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3178816, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 32768, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 15, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145712, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393984, 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:39:55.081 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318867439616, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 33554432, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 395448, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:55.082 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 15388, 2020-03-11T22:39:55.083 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221210084 2020-03-11T22:39:55.083 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.083 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:39:55.083 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:55.083 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:55.084 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.085 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 47947, 2020-03-11T22:39:55.086 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:55.087 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:55.088 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:55.089 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:55.090 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:55.091 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19529728, 2020-03-11T22:39:55.092 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 142522, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.093 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 17, 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 17, 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 2, 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 2, 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.094 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 146028888083 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672974 2020-03-11T22:39:55.095 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 150323855379 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.096 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 1 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:55.097 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.098 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 1, 2020-03-11T22:39:55.099 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:39:55.100 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:37:42.115935+0000", 2020-03-11T22:39:55.101 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 380, 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:39:55.102 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:39:55.103 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:39:55.103 INFO:tasks.workunit.client.0.smithi118.stderr:report 1012384831 2020-03-11T22:39:55.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:55.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:39:55.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:55.104 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery 2020-03-11T22:39:55.273 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:55.274 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:43.800351+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:39:55.274 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:39:55.274 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:55.275 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:39:55.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:39:55.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:39:55.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:39:55.285 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:55.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:39:55.375 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:55.382 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:39:55.383 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:39:55.383 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:39:55.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:55.385 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:39:55.474 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:55.481 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:39:55.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:55.482 INFO:tasks.workunit.client.0.smithi118.stdout:revive divergent 1 2020-03-11T22:39:55.800 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:55.800 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:43.800351+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:39:55.800 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:39:55.800 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:55.801 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:39:55.810 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:39:56.437 INFO:tasks.workunit.client.0.smithi118.stderr:noup is set 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:56.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:56.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:56.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:56.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:56.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:56.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:56.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:56.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:56.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:56.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:56.454 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:56.454 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:56.454 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:56.454 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:56.455 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:56.455 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:39:56.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:39:56.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:39:56.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:56.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:56.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:56.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:56.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:56.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:56.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:56.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:56.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:56.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:39:56.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:56.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:39:56.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:39:56.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:56.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:56.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:56.459 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:39:56.475 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:39:56.471+0000 7fd3093d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:56.476 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:39:56.473+0000 7fd3093d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:56.478 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:39:56.475+0000 7fd3093d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:56.796 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:39:56.796 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:39:56.994 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:39:56.991+0000 7fd3093d0ec0 -1 Falling back to public interface 2020-03-11T22:39:57.922 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:39:57.918+0000 7fd3093d0ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:40:01.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:40:01.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:01.797 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery divergent 2020-03-11T22:40:02.125 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:02.125 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:43.800351+0000 38'101 38:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:02.126 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:02.126 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:02.127 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:02.135 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:40:02.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:40:02.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:40:02.136 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:02.136 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:02.136 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:02.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:40:02.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:02.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:40:02.227 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:40:02.234 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:40:03.010 INFO:tasks.workunit.client.0.smithi118.stderr:noup is unset 2020-03-11T22:40:03.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:40:03.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:03.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:03.020 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:03.020 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:03.389 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30238,v1:127.0.0.1:6803/30238] [v2:127.0.0.1:6804/30238,v1:127.0.0.1:6805/30238] exists,up 847b0c18-b574-4952-9cbd-7148f90e9b63 2020-03-11T22:40:03.389 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:03.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:03.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:03.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:03.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:03.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:03.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:03.728 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33015,v1:127.0.0.1:6819/33015] [v2:127.0.0.1:6820/33015,v1:127.0.0.1:6821/33015] exists,up aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:40:03.729 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:03.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:03.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:03.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:03.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:03.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:03.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:03.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:04.070 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30432,v1:127.0.0.1:6811/30432] [v2:127.0.0.1:6812/30432,v1:127.0.0.1:6813/30432] exists,up d0bbeb21-437a-424d-9469-615d7849a1d4 2020-03-11T22:40:04.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:04.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:04.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:04.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:04.390 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:04.391 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:40:02.880243+0000 28'106 42:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:04.391 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:04.391 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:04.392 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:04.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:40:04.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:04.401 INFO:tasks.workunit.client.0.smithi118.stdout:wait for peering 2020-03-11T22:40:04.720 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:04.720 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:40:02.880243+0000 28'106 42:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:04.721 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:04.721 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:04.721 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:04.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:40:11.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:40:11.846 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:11.846 INFO:tasks.workunit.client.0.smithi118.stdout:killing divergent 1 2020-03-11T22:40:12.177 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:12.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:40:03.285463+0000 38'101 44:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:12.177 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:12.177 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:12.178 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:40:12.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:12.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:12.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:12.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17426.out 2020-03-11T22:40:12.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17426.out 2020-03-11T22:40:12.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:12.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:12.294 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:12.294 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:12.294 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:12.294 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17426.out 2020-03-11T22:40:12.861 INFO:tasks.workunit.client.0.smithi118.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=42/44 n=101 ec=22/22 lis/c=42/22 les/c/f=44/23/0 sis=42 pruub=8.518778967s) 2020-03-11T22:40:12.877 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:12.909 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:12.927 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:12.950 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:40:12.961 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:40:12.973 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:40:12.978 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:40:12.985 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:40:13.008 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:40:13.028 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:40:13.036 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:40:13.037 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:40:13.041 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:40:13.041 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:40:13.047 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:40:13.054 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:40:13.054 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:40:13.054 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:40:13.055 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:40:13.057 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:40:13.066 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:40:13.067 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:40:13.071 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:40:13.080 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:40:13.082 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:40:13.083 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:40:13.083 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:40:13.084 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:40:13.084 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:40:13.085 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:40:13.090 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:40:13.091 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:40:13.091 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:40:13.092 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:40:13.092 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:40:13.094 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:40:13.102 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:40:13.103 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:40:13.104 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:40:13.104 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:40:13.107 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:40:13.109 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:40:13.109 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:40:13.115 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:40:13.116 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:40:13.121 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:40:13.122 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:40:13.123 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:40:13.124 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:40:13.124 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:40:13.125 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:40:13.125 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:40:13.131 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:40:13.133 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:40:13.134 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:40:13.135 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:40:13.135 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:40:13.135 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:40:13.136 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:40:13.136 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:40:13.137 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:40:13.138 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:40:13.138 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:40:13.138 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:40:13.138 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:40:13.139 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:40:13.139 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:40:13.140 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:40:13.142 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:40:13.142 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:40:13.142 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:40:13.143 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:40:13.150 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:40:13.150 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:40:13.152 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:40:13.152 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:40:13.152 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:40:13.153 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:40:13.153 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:40:13.153 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:40:13.154 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:40:13.154 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:40:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:40:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:40:13.156 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:40:13.156 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:40:13.157 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:40:13.157 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:40:13.158 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:40:13.158 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:40:13.158 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:40:13.159 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:40:13.159 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:40:13.160 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:40:13.161 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:40:13.161 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:40:13.162 INFO:tasks.workunit.client.0.smithi118.stderr:Export successful 2020-03-11T22:40:13.162 INFO:tasks.workunit.client.0.smithi118.stdout: marking collection for removal 2020-03-11T22:40:13.162 INFO:tasks.workunit.client.0.smithi118.stdout:setting '_remove' omap key 2020-03-11T22:40:13.162 INFO:tasks.workunit.client.0.smithi118.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:40:13.240 INFO:tasks.workunit.client.0.smithi118.stderr:Remove successful 2020-03-11T22:40:13.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17426.out 2020-03-11T22:40:13.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:13.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:13.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:13.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:13.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:13.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17426.out 2020-03-11T22:40:14.097 INFO:tasks.workunit.client.0.smithi118.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:40:14.098 INFO:tasks.workunit.client.0.smithi118.stdout:Importing pgid 1.0 2020-03-11T22:40:14.098 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:14.098 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.189 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:14.189 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.197 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:14.197 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.205 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:40:14.205 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.229 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:40:14.229 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.247 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:40:14.247 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.264 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:40:14.264 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.272 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:40:14.272 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.280 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:40:14.280 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.289 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:40:14.289 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.297 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:40:14.297 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.306 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:40:14.306 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.314 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:40:14.314 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.322 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:40:14.322 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.354 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:40:14.354 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.361 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:40:14.361 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.370 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:40:14.370 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.378 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:40:14.378 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.386 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:40:14.386 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.395 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:40:14.395 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.403 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:40:14.403 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.411 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:40:14.412 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.421 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:40:14.421 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.428 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:40:14.428 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.436 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:40:14.437 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.445 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:40:14.445 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.475 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:40:14.475 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.487 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:40:14.487 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.521 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:40:14.521 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.530 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:40:14.531 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.539 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:40:14.539 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.547 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:40:14.547 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.555 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:40:14.556 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.564 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:40:14.564 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.572 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:40:14.572 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.581 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:40:14.581 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.589 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:40:14.589 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.597 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:40:14.597 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.606 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:40:14.606 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.614 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:40:14.614 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.622 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:40:14.622 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.721 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:40:14.722 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:40:14.731 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.739 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:40:14.739 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.747 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:40:14.748 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.756 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:40:14.756 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.764 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:40:14.764 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.772 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:40:14.773 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.781 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:40:14.781 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.811 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:40:14.811 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.823 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:40:14.823 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.831 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:40:14.831 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.839 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:40:14.839 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.847 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:40:14.848 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.856 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:40:14.856 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.896 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:40:14.897 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.907 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:40:14.907 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.915 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:40:14.915 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.923 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:40:14.923 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.932 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:40:14.932 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.940 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:40:14.940 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.948 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:40:14.949 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.957 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:40:14.957 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.965 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:40:14.965 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:14.974 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:40:14.974 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.007 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:40:15.008 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.040 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:40:15.041 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.048 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:40:15.049 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.080 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:40:15.080 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.090 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:40:15.090 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.098 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:40:15.099 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.107 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:40:15.107 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.115 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:40:15.116 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.157 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:40:15.158 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.165 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:40:15.165 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.174 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:40:15.174 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.182 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:40:15.182 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.190 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:40:15.191 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.199 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:40:15.199 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.207 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:40:15.207 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.215 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:40:15.215 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.224 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:40:15.224 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.255 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:40:15.255 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.265 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:40:15.266 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.274 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:40:15.274 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.282 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:40:15.282 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.291 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:40:15.291 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.299 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:40:15.299 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.307 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:40:15.307 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.316 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:40:15.316 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.324 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:40:15.324 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.332 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:40:15.332 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.341 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:40:15.341 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.358 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:40:15.358 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.366 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:40:15.366 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.397 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:40:15.397 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:40:15.407 INFO:tasks.workunit.client.0.smithi118.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=42/44 n=101 ec=22/22 lis/c=42/22 les/c/f=44/23/0 sis=42 pruub=8.518778967s) 2020-03-11T22:40:15.416 INFO:tasks.workunit.client.0.smithi118.stdout:Import successful 2020-03-11T22:40:15.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:40:15.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:15.942 INFO:tasks.workunit.client.0.smithi118.stdout:reviving divergent 1 2020-03-11T22:40:16.273 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:16.273 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:40:03.285463+0000 38'101 44:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:16.273 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:16.273 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:16.274 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:16.285 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:16.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:16.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:16.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:16.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:16.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:40:16.291 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:40:16.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:40:16.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:16.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:16.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:16.312 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:16.309+0000 7f735bc88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.313 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:16.311+0000 7f735bc88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.314 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:16.311+0000 7f735bc88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.628 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:16.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:16.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:16.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:16.630 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:16.829 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:16.826+0000 7f735bc88ec0 -1 Falling back to public interface 2020-03-11T22:40:16.974 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33015,v1:127.0.0.1:6819/33015] [v2:127.0.0.1:6820/33015,v1:127.0.0.1:6821/33015] exists,up aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:40:16.974 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:16.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:16.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:16.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:16.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:16.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:16.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:17.321 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33015,v1:127.0.0.1:6819/33015] [v2:127.0.0.1:6820/33015,v1:127.0.0.1:6821/33015] exists,up aa764004-f9e8-4507-a028-b30c8088b6f2 2020-03-11T22:40:17.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:17.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:17.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:17.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:40:17.635 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:17.633+0000 7f735bc88ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:40:37.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:40:37.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:40:37.323 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:40:37.324 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:37.324 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:37.324 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:37.325 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:40:37.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:37.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:40:37.419 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:40:37.419 INFO:tasks.workunit.client.0.smithi118.stdout: "ops": [], 2020-03-11T22:40:37.419 INFO:tasks.workunit.client.0.smithi118.stdout: "num_ops": 0 2020-03-11T22:40:37.419 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:40:37.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:40:37.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:37.427 INFO:tasks.workunit.client.0.smithi118.stdout:allowing recovery 2020-03-11T22:40:37.749 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:37.749 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:40:19.009870+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:37.749 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:37.749 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:37.750 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:37.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:37.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:40:37.878 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:37.886 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:37.886 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:40:38.007 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:38.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:38.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:40:38.135 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:38.143 INFO:tasks.workunit.client.0.smithi118.stdout:reading divergent objects 2020-03-11T22:40:38.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:40:38.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:38.463 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:38.464 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:40:19.009870+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:03.903397+0000 0'0 2020-03-11T22:38:03.903397+0000 0 2020-03-11T22:40:38.464 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:40:38.464 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:38.465 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:40:38.473 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:40:38.475 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:40:38.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:38.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:40:38.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:38.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:40:38.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:38.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:40:38.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:38.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:40:38.587 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:40:38.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:38.588 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:40:38.614 INFO:tasks.workunit.client.0.smithi118.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:40:38.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:38.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:40:38.642 INFO:tasks.workunit.client.0.smithi118.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:40:38.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:40:38.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:38.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:38.648 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'105 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:38:25.187974+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:38.648 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'105 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:38:25.187974+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: more recent entry found: 38'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:39:20.179460+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4533.0:1 2020-03-11T22:38:25.183490+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4533.0:1 2020-03-11T22:38:25.183490+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:38.649 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:38:25.187826+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:38:25.187826+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:40:38.650 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:38:25.187826+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:38:25.172318+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.651 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:38:25.172318+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:38:25.172318+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:40:38.652 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:40:38.653 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4526.0:1 2020-03-11T22:38:25.172567+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.653 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4526.0:1 2020-03-11T22:38:25.172567+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.653 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4526.0:1 2020-03-11T22:38:25.172567+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.654 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4532.0:1 2020-03-11T22:38:25.182226+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4532.0:1 2020-03-11T22:38:25.182226+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:40:38.655 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:03.238+0000 7fd2e758b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:40:38.656 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:38.656 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:38.656 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:38.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:38.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:40:38.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17426.out 2020-03-11T22:40:38.657 INFO:tasks.workunit.client.0.smithi118.stdout:success 2020-03-11T22:40:38.658 INFO:tasks.workunit.client.0.smithi118.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17426.out': No such file or directory 2020-03-11T22:40:38.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:40:38.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:40:38.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:40:39.098 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' does not exist 2020-03-11T22:40:39.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:40:39.106 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:39.106 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:39.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:39.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:39.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:39.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:39.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:40:39.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:39.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:39.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:39.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:39.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:39.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:39.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:39.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:39.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:39.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:39.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:39.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:39.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:39.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:39.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:39.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:39.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:39.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:39.223 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:39.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:39.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:39.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:39.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:39.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:40:39.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:39.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:39.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:39.240 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:39.241 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:39.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:39.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:39.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:39.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:39.242 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:39.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:39.242 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:39.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:39.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:39.243 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:39.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:39.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:39.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:39.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:39.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:39.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:39.247 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:39.248 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:39.248 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.248 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.248 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:40:39.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:39.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:39.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:40:39.250 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:40:39.250 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.250 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.250 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17426 2020-03-11T22:40:39.251 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:40:39.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:40:39.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:40:39.253 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:40:39.254 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:40:39.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:40:39.254 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:40:39.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:40:39.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:39.554 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:39.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:40:39.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:40:39.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:40:39.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:39.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:39.733 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.734 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:39.735 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get fsid 2020-03-11T22:40:39.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:40:39.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:39.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:39.818 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:39.819 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:39.819 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:40:39.819 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:39.819 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get mon_host 2020-03-11T22:40:39.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:40:39.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:40:39.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:40:39.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:40:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:40:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:40:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:40:40.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:40:40.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:40.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:40.771 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:40.772 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:40.772 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:40.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:40.772 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:40.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:40.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:40.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:40.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:40.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:40.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:40.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:40.805 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:40.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:40:40.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 b7cca827-16ff-4908-ab2c-ccbfc3263ec2' 2020-03-11T22:40:40.806 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:40.806 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:40:40.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDoaGlex5KyMBAA5IB+Bq/s/CK0WklJkK7LKQ== 2020-03-11T22:40:40.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDoaGlex5KyMBAA5IB+Bq/s/CK0WklJkK7LKQ=="}' 2020-03-11T22:40:40.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b7cca827-16ff-4908-ab2c-ccbfc3263ec2 -i td/divergent-priors/0/new.json 2020-03-11T22:40:41.144 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:41.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:40:41.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDoaGlex5KyMBAA5IB+Bq/s/CK0WklJkK7LKQ== --osd-uuid b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:40:41.183 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.179+0000 7fc12ed23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.184 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.181+0000 7fc12ed23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.186 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.183+0000 7fc12ed23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.187 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.183+0000 7fc12ed23ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:41.187 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.183+0000 7fc12ed23ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:41.187 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.183+0000 7fc12ed23ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:41.229 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:41.225+0000 7fc12ed23ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:40:44.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:40:44.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:44.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:40:44.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:44.255 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:40:44.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:40:44.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:44.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:44.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:44.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:44.702 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:40:44.726 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:44.722+0000 7f0113e9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:44.728 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:44.724+0000 7f0113e9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:44.730 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:44.727+0000 7f0113e9eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:45.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:40:45.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:45.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:45.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:45.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:45.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:45.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:45.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:45.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:45.093 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:45.245 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:45.242+0000 7f0113e9eec0 -1 Falling back to public interface 2020-03-11T22:40:45.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:46.012 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:46.008+0000 7f0113e9eec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:40:46.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:46.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:46.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:46.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:46.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:46.452 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:40:46.797 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34513,v1:127.0.0.1:6803/34513] [v2:127.0.0.1:6804/34513,v1:127.0.0.1:6805/34513] exists,up b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:40:46.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:46.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:46.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:46.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:46.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:46.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:46.805 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:46.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:46.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:46.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:40:46.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 aee7f2fc-043b-4c33-a93a-605dc6e31b60' 2020-03-11T22:40:46.808 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:46.808 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:40:46.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDuaGleTUqjMBAAnviVIyb+ynWqwA4u7Tri5w== 2020-03-11T22:40:46.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDuaGleTUqjMBAAnviVIyb+ynWqwA4u7Tri5w=="}' 2020-03-11T22:40:46.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new aee7f2fc-043b-4c33-a93a-605dc6e31b60 -i td/divergent-priors/1/new.json 2020-03-11T22:40:47.375 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:40:47.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:40:47.386 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDuaGleTUqjMBAAnviVIyb+ynWqwA4u7Tri5w== --osd-uuid aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:40:47.414 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.410+0000 7f4efc5eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:47.416 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.412+0000 7f4efc5eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:47.418 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.414+0000 7f4efc5eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:47.418 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.414+0000 7f4efc5eaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:47.419 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.414+0000 7f4efc5eaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:47.419 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.414+0000 7f4efc5eaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:47.495 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:47.492+0000 7f4efc5eaec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:40:50.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:40:50.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:50.522 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:40:50.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:40:50.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:50.975 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:40:50.975 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:40:50.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:50.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:50.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:50.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:51.002 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:50.999+0000 7f3dc9220ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:51.003 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:51.001+0000 7f3dc9220ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:51.005 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:51.002+0000 7f3dc9220ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:51.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:51.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:51.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:51.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:51.322 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:51.520 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:51.517+0000 7f3dc9220ec0 -1 Falling back to public interface 2020-03-11T22:40:51.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:52.292+0000 7f3dc9220ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:52.689 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:40:53.055 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34868,v1:127.0.0.1:6811/34868] [v2:127.0.0.1:6812/34868,v1:127.0.0.1:6813/34868] exists,up aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:40:53.056 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:40:53.056 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:53.056 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:53.056 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:53.056 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:53.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:53.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:53.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:53.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:53.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:53.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:53.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:53.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:53.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 95b460b2-26b5-4e83-a391-67730479ac69' 2020-03-11T22:40:53.062 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:53.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD1aGleMKQ/BBAAplzritZ+j/4Q/OeD6ZJIXw== 2020-03-11T22:40:53.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD1aGleMKQ/BBAAplzritZ+j/4Q/OeD6ZJIXw=="}' 2020-03-11T22:40:53.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 95b460b2-26b5-4e83-a391-67730479ac69 -i td/divergent-priors/2/new.json 2020-03-11T22:40:53.586 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:40:53.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:40:53.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1aGleMKQ/BBAAplzritZ+j/4Q/OeD6ZJIXw== --osd-uuid 95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:40:53.625 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.622+0000 7f6b4129fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:53.626 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.624+0000 7f6b4129fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:53.628 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.626+0000 7f6b4129fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:53.629 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.626+0000 7f6b4129fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:53.629 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.626+0000 7f6b4129fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:53.629 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.626+0000 7f6b4129fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:53.678 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:53.675+0000 7f6b4129fec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:40:56.796 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:40:56.796 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:56.798 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:40:56.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:40:56.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:57.252 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:40:57.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:40:57.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:57.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:57.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:57.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:57.279 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:57.276+0000 7fda4ab90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:57.280 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:57.278+0000 7fda4ab90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:57.282 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:57.280+0000 7fda4ab90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:57.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:40:57.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:57.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:57.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:57.598 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:40:57.797 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:57.794+0000 7fda4ab90ec0 -1 Falling back to public interface 2020-03-11T22:40:57.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:58.553 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:40:58.551+0000 7fda4ab90ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:40:58.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:58.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:58.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:58.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:58.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:58.943 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:40:59.319 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35223,v1:127.0.0.1:6819/35223] [v2:127.0.0.1:6820/35223,v1:127.0.0.1:6821/35223] exists,up 95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:40:59.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:59.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:59.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:59.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:41:00.039 INFO:tasks.workunit.client.0.smithi118.stderr:noout is set 2020-03-11T22:41:00.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:41:00.841 INFO:tasks.workunit.client.0.smithi118.stderr:noin is set 2020-03-11T22:41:00.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:41:01.605 INFO:tasks.workunit.client.0.smithi118.stderr:nodown is set 2020-03-11T22:41:01.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:41:01.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:41:02.156 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' already exists 2020-03-11T22:41:02.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:41:03.165 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:41:03.805 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 3 2020-03-11T22:41:03.813 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:41:04.456 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 min_size to 2 2020-03-11T22:41:04.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:41:05.125 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:41:05.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:41:05.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:05.133 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:05.476 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:05.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:41:05.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:41:05.605 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:05.605 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:05.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:41:05.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:41:05.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:05.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:05.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509445 2020-03-11T22:41:05.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509445' 2020-03-11T22:41:05.864 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:05.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:41:05.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:05.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:05.866 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:41:05.866 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:05.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:41:05.867 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:41:05.867 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:05.867 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:41:06.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:41:06.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:06.210 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:41:06.210 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:06.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:06.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:41:06.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:06.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:41:06.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:41:06.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:06.213 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:41:06.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:41:06.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:06.579 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509445 2020-03-11T22:41:06.579 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:06.579 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:06.580 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509445 2020-03-11T22:41:06.580 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:06.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509445 2020-03-11T22:41:06.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2020-03-11T22:41:06.581 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:06.581 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509445 2020-03-11T22:41:06.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509445 -lt 64424509445 2020-03-11T22:41:06.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:41:06.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:06.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:06.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:06.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:07.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:07.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:07.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:07.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:07.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:07.357 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:07.357 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:41:07.358 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:41:07.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:07.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:07.358 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:07.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:41:07.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:41:07.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:07.487 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:07.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:41:07.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:41:07.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:07.654 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:07.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:41:07.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509447' 2020-03-11T22:41:07.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:07.784 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:41:07.784 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:07.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:07.785 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:41:07.786 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:07.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:41:07.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:41:07.787 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:07.787 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:41:08.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:41:08.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:08.153 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:41:08.153 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:08.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:08.154 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:41:08.154 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:08.156 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:41:08.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672967 -lt 42949672967 2020-03-11T22:41:08.500 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:08.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:41:08.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:08.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:08.501 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:41:08.501 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:08.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:41:08.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:41:08.503 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:08.503 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:41:08.872 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509447 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:09.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:09.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:09.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:09.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:09.320 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:09.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:09.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:09.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:09.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:09.648 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:09.648 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:09.648 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:10.088 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:10.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:10.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:10.089 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:41:10.089 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:41:10.411 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:41:10.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:41:10.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:41:10.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:10.420 INFO:tasks.workunit.client.0.smithi118.stdout:primary and soon to be divergent is 1 2020-03-11T22:41:10.739 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:10.739 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:41:03.434072+0000 0'0 24:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:41:10.739 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:41:10.739 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:10.740 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:41:10.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:41:10.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:10.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:41:10.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:41:10.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:41:10.752 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:41:10.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:10.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:41:10.752 INFO:tasks.workunit.client.0.smithi118.stdout:writing initial objects 2020-03-11T22:41:10.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:10.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:41:10.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:10.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:41:10.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:10.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:41:10.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:10.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:41:11.091 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.091 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:41:11.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:41:11.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:41:11.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:41:11.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:41:11.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:41:11.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:41:11.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:41:11.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:41:11.661 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.661 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:41:11.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:41:11.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:41:11.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:41:11.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:41:11.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:11.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:41:12.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:41:12.120 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.120 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:41:12.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:41:12.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:41:12.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:41:12.357 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.357 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:41:12.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:41:12.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:41:12.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:41:12.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:41:12.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:41:12.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:41:12.790 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.790 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:41:12.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:41:12.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:41:12.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:41:12.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:12.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:41:13.055 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.055 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:41:13.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:41:13.172 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.172 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:41:13.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:41:13.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:41:13.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:41:13.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.468 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:41:13.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:41:13.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:41:13.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:41:13.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:41:13.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:41:13.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:41:13.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:41:13.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:13.971 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:41:14.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:41:14.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.070 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:41:14.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:41:14.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:41:14.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:41:14.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:41:14.472 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.472 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:41:14.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:41:14.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:41:14.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:41:14.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.647 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:41:14.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:41:14.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:41:14.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:41:14.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:41:14.971 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:14.971 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:41:15.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:41:15.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:41:15.229 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.229 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:41:15.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:41:15.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:41:15.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:41:15.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:41:15.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:41:15.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:41:15.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:41:15.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:41:15.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.813 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:41:15.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:41:15.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:41:15.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:15.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:41:16.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:41:16.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:41:16.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:41:16.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:41:16.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:41:16.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:41:16.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:41:16.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:41:16.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:41:16.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:41:16.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:41:16.979 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:16.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:41:17.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:17.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:41:17.072 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:17.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:41:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:41:17.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:17.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:41:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:41:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:17.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:17.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:17.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:17.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:17.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:17.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:17.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:17.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:17.615 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:41:17.615 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:41:17.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:17.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:17.616 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:17.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:41:17.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:41:17.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:17.767 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:17.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:41:17.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2020-03-11T22:41:17.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:17.896 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:18.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:41:18.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509451' 2020-03-11T22:41:18.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:18.025 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:41:18.025 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:18.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:18.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:41:18.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:18.027 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:41:18.027 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:41:18.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:18.028 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:41:18.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:41:18.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:18.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:41:18.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:18.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:18.369 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:41:18.369 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:18.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:41:18.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:41:18.370 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:18.370 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:41:18.713 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672971 2020-03-11T22:41:18.713 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:18.714 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:41:18.714 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:18.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:18.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:41:18.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:18.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:41:18.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:41:18.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:18.716 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:41:19.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509451 2020-03-11T22:41:19.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:20.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:20.060 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:20.411 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509452 -lt 64424509451 2020-03-11T22:41:20.412 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:20.412 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:20.412 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:20.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:20.877 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:20.877 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:20.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:20.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:20.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:20.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:21.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:21.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:21.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:21.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:21.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:21.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:21.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:21.650 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:41:21.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:41:21.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:41:21.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:41:21.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:41:21.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:41:21.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:41:21.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:21.994 INFO:tasks.workunit.client.0.smithi118.stdout:blackholing osds 0 2 2020-03-11T22:41:22.316 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:22.316 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:41:03.434072+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:41:22.316 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:41:22.317 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:22.317 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:41:22.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:22.327 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:41:22.328 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:41:22.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:22.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:41:22.413 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:41:22.413 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:41:22.414 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:41:22.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:22.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:41:22.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:22.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:22.422 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:22.423 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:22.423 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:41:22.423 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:41:22.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:22.424 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:41:22.508 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:41:22.508 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:41:22.508 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:41:22.515 INFO:tasks.workunit.client.0.smithi118.stdout:create new divergent objects 2020-03-11T22:41:22.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:41:22.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:41:22.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:22.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:22.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:41:22.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:41:22.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:41:22.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:41:22.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:41:22.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:41:22.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:41:22.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:41:22.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:41:22.528 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:41:22.528 INFO:tasks.workunit.client.0.smithi118.stdout:writing divergent objects 2020-03-11T22:41:22.529 INFO:tasks.workunit.client.0.smithi118.stdout:remove divergent objects 2020-03-11T22:41:22.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:41:22.532 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:41:22.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:41:22.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:41:22.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:41:22.533 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:41:22.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:41:22.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:41:22.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:41:32.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:41:32.528 INFO:tasks.workunit.client.0.smithi118.stderr:rados: no process found 2020-03-11T22:41:32.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:41:32.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:32.528 INFO:tasks.workunit.client.0.smithi118.stdout:killing all the osds 2020-03-11T22:41:32.854 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:32.854 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:41:03.434072+0000 29'106 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:41:32.854 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:41:32.855 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:32.856 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:41:32.864 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:41:32.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:32.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:32.865 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:32.865 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:32.865 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:32.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:32.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:32.971 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:41:33.456 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already down. 2020-03-11T22:41:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:33.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:41:33.991 INFO:tasks.workunit.client.0.smithi118.stderr:osd.1 is already down. 2020-03-11T22:41:33.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:33.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:41:34.500 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already down. 2020-03-11T22:41:34.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:34.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:41:35.024 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already out. 2020-03-11T22:41:35.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:35.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:41:35.552 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already out. 2020-03-11T22:41:35.560 INFO:tasks.workunit.client.0.smithi118.stdout:bringing up non_divergent 0 2 2020-03-11T22:41:35.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:41:35.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:35.878 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:35.878 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:41:03.434072+0000 29'106 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:41:35.878 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:41:35.878 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:35.879 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:41:35.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:35.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:41:35.916 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:35.914+0000 7fea578c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:35.918 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:35.916+0000 7fea578c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:35.920 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:35.918+0000 7fea578c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:36.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:36.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:36.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:36.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:36.269 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:41:36.439 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:36.436+0000 7fea578c0ec0 -1 Falling back to public interface 2020-03-11T22:41:36.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:37.311 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:37.309+0000 7fea578c0ec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:41:37.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:37.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:37.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:37.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:37.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:37.612 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:41:37.993 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/39222,v1:127.0.0.1:6803/39222] [v2:127.0.0.1:6804/39222,v1:127.0.0.1:6805/39222] exists,up b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:41:37.993 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:41:37.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:37.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:37.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:41:37.997 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:37.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:41:37.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:41:38.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:38.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:38.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:38.018 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:38.016+0000 7f8c423f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:38.020 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:38.017+0000 7f8c423f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:38.021 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:38.019+0000 7f8c423f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:38.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:38.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:38.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:38.356 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:38.356 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:41:38.536 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:38.533+0000 7f8c423f0ec0 -1 Falling back to public interface 2020-03-11T22:41:38.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:39.430 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:41:39.427+0000 7f8c423f0ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:41:39.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:39.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:39.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:39.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:39.699 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:41:39.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:40.046 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/39418,v1:127.0.0.1:6811/39418] [v2:127.0.0.1:6812/39418,v1:127.0.0.1:6813/39418] exists,up 95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:41:40.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:40.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:40.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:40.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:40.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:41:40.628 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already in. 2020-03-11T22:41:40.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:40.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:41:41.104 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already in. 2020-03-11T22:41:41.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:41.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:41:41.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:41.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:41.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:41.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:41.114 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:41.115 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:41.115 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:41.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:41.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:41.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:41.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:41.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:41.171 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:41.171 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:41.171 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:41.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:41.549 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:41:41.549 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:41:41.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:41.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:41.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:41.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:41:41.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:41:41.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:41.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:41.768 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:41:41.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:41:41.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:41:41.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:41.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:41.905 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:41:41.905 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:41:41.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:41.906 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:41:41.906 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:41.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:41.907 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:41:41.907 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:41.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:41:41.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:41:41.908 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:41.909 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:41:42.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:41:42.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:42.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:41:42.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:42.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:42.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:41:42.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:42.285 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:41:42.285 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:41:42.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:42.286 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:41:42.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:41:42.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:41:42.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:42.627 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:41:42.627 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:42.627 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:42.628 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:41:42.628 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:42.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:41:42.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:41:42.629 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:42.629 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:41:42.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:41:42.968 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:42.969 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:42.969 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:43.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:43.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:43.413 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:43.413 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:43.413 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:43.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:43.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:43.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:43.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:43.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:43.743 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:43.743 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:44.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:44.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:41:44.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:41:44.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:41:44.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:44.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:44.290 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:44.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:44.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:44.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:44.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:44.623 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:44.623 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:45.064 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:45.065 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:45.065 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:45.065 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:45.065 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:45.065 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:45.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:45.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:45.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:45.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:45.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:45.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:41:45.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:45.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:41:45.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:45.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:45.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:45.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:45.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:45.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:45.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:45.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:46.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:46.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:46.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:46.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:46.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:46.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:46.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:47.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:47.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:47.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:47.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:47.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:47.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:47.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:47.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:47.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:47.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:47.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:47.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:48.195 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:48.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:48.197 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:48.197 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:48.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:48.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:48.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:41:48.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:48.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:41:49.437 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:49.437 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:49.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:49.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:49.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:49.439 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:49.439 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:49.439 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:49.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:49.780 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:49.780 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:49.780 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:50.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:50.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:50.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:50.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:50.684 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:52.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:52.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:52.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:52.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:52.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:52.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:52.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:52.286 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:52.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:52.633 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:52.633 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:52.633 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:53.072 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:53.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:53.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:53.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:53.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:53.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:53.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:53.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:53.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:56.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:56.717 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:56.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:57.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:57.058 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:57.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:57.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:57.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:57.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:57.499 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:57.499 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:57.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:57.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:57.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:57.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:57.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:42:04.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:04.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:04.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:04.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:04.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:04.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:04.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:04.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:04.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:04.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:04.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:04.681 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:05.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:05.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:05.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:05.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:05.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:05.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:05.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:05.118 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:05.118 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:05.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:05.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:05.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:42:05.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:05.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:42:12.860 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:12.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:12.862 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:12.862 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:12.862 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:12.862 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:12.863 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:12.863 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:13.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:13.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:13.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:13.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:13.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:14.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:14.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:14.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:42:14.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:42:14.541 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:42:14.541 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "b09154b0-6144-4fd9-8d95-50275240a02f", 2020-03-11T22:42:14.541 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:42:14.538063+0000", 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:42:14.542 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 32 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.546 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.547 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.548 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.549 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.550 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.551 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.552 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.553 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.554 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.555 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.556 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.557 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.558 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:14.559 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:14.560 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.561 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.562 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:14.563 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:40:39.273772Z", 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:40:39.273772Z", 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:42:14.564 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:42:14.565 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.566 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:42:14.567 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:42:14.568 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 42, 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:40:40.008033+0000", 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:41:46.344662+0000", 2020-03-11T22:42:14.569 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:41:39.502268+0000", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:41:40.918578+0000", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:42:14.570 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 7, 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:42:14.571 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:42:14.572 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:41:01.865776+0000", 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 1, 2020-03-11T22:42:14.573 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool", 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 0, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 32, 2020-03-11T22:42:14.574 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 32, 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 32, 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 32, 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "41", 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "41", 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 0, 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 0, 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [], 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "", 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 0, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "b7cca827-16ff-4908-ab2c-ccbfc3263ec2", 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 29, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 35, 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 41, 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.587 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:42:14.588 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.589 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.590 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.591 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/39222", 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/39222", 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39222", 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39222", 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "aee7f2fc-043b-4c33-a93a-605dc6e31b60", 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:14.593 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 22, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:42:14.594 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.595 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.596 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.597 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:14.598 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.599 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:14.600 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.601 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/34868", 2020-03-11T22:42:14.602 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/34868", 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34868", 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34868", 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.603 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "95b460b2-26b5-4e83-a391-67730479ac69", 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:14.604 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 29, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 36, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 32, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.605 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.606 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.607 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.608 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.609 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.610 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.611 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.612 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.613 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/39418", 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/39418", 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39418", 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39418", 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:42:14.614 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:14.615 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:33.264233+0000", 2020-03-11T22:42:14.616 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:14.616 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:14.616 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:14.616 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:14.616 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:46.983698+0000", 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:33.758590+0000", 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:14.617 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:53.341593+0000", 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:14.618 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:34.296860+0000", 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:59.579706+0000", 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:14.619 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:42:14.620 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.621 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:42:14.622 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/39222,v1:127.0.0.1:6805/39222]", 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:14.623 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:42:14.624 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:14.625 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:14.626 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/39222,v1:127.0.0.1:6803/39222]", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39222,v1:127.0.0.1:6809/39222]", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39222,v1:127.0.0.1:6807/39222]", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:14.627 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:14.628 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:14.629 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/34868,v1:127.0.0.1:6813/34868]", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:14.630 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:14.631 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:14.632 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/34868,v1:127.0.0.1:6811/34868]", 2020-03-11T22:42:14.633 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34868,v1:127.0.0.1:6817/34868]", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34868,v1:127.0.0.1:6815/34868]", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:14.634 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:14.635 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/39418,v1:127.0.0.1:6813/39418]", 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:14.636 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:14.637 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:14.638 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:14.639 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/39418,v1:127.0.0.1:6811/39418]", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39418,v1:127.0.0.1:6817/39418]", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39418,v1:127.0.0.1:6815/39418]", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:14.640 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:14.641 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 42, 2020-03-11T22:42:14.642 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.643 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:14.644 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:42:14.645 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:42:14.646 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.647 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.648 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:42:14.649 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:42:14.650 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:14.651 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:42:14.652 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.653 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:42:14.654 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:14.655 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.656 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:14.657 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.659 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:42:14.660 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:14.661 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:14.662 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:42:14.663 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.664 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:42:14.665 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:42:14.666 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.667 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:42:14.668 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:42:14.669 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 0, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:42:14.670 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:42:14.671 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:42:14.672 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:42:14.673 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:42:14.673 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:42:14.673 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:42:14.673 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:42:14.674 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.674 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.674 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:42:14.674 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:42:14.674 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:42:14.675 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.675 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:42:14.675 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:42:14.675 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.676 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:42:14.676 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:42:14.676 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:42:14.676 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 32, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 32, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:42:14.677 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:42:14.678 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:14.679 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:42:14.679 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:14.679 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:14.679 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:14.679 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:14.680 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:42:14.680 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:42:14.680 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:42:14.680 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:42:14.680 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:14.681 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:42:14.682 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:14.683 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:42:14.684 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:14.685 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 512, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 512, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 64, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 64, 2020-03-11T22:42:14.686 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 65, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:14.687 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179968, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33920, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 13, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145714, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311392832, 2020-03-11T22:42:14.688 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866259968, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34734080, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 423129, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:14.689 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:14.690 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:14.690 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 13558, 2020-03-11T22:42:14.690 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221211914 2020-03-11T22:42:14.690 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.690 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.691 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:42:14.692 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 65, 2020-03-11T22:42:14.693 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179968, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33920, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 13, 2020-03-11T22:42:14.694 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145714, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311392832, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866259968, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34734080, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 423129, 2020-03-11T22:42:14.695 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 13558, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221211914 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:14.696 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.697 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.698 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 32, 2020-03-11T22:42:14.699 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:42:14.700 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48300, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:42:14.701 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:42:14.702 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 100, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 100, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:14.703 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:42:14.704 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:14.705 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.706 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19529728, 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 143228, 2020-03-11T22:42:14.707 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.708 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 512, 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 512, 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 64, 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 64, 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.709 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 150323855370 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.710 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672974 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 154618822665 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.711 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 32 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:42:14.712 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 32, 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.713 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:14.714 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 32, 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 32, 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:14.715 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:40:40.008325+0000", 2020-03-11T22:42:14.716 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 307, 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:42:14.717 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:14.718 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:42:14.719 INFO:tasks.workunit.client.0.smithi118.stderr:report 1210412217 2020-03-11T22:42:14.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:42:14.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:42:14.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:42:14.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:42:14.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:14.720 INFO:tasks.workunit.client.0.smithi118.stdout:writing non-divergent object existing_4 2020-03-11T22:42:14.872 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.873 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.874 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.875 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.876 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:14.877 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:14.878 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:14.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:14.928 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:14.928 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:14.929 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:14.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:14.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:14.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:14.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:14.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:14.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:14.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:14.985 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:15.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:15.325 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:42:15.326 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:42:15.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:15.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:15.326 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:15.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:42:15.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372' 2020-03-11T22:42:15.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:15.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:15.544 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:42:15.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:42:15.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372 1-' 2020-03-11T22:42:15.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:15.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:15.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:42:15.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372 1- 2-154618822668' 2020-03-11T22:42:15.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:15.682 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855372 2020-03-11T22:42:15.683 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:15.684 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:15.684 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855372 2020-03-11T22:42:15.685 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:15.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:42:15.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855372' 2020-03-11T22:42:15.685 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:15.686 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 150323855372 2020-03-11T22:42:16.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855370 -lt 150323855372 2020-03-11T22:42:16.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:17.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:17.023 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:17.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855372 -lt 150323855372 2020-03-11T22:42:17.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:17.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:42:17.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:17.372 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:17.373 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:42:17.373 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:17.373 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:42:17.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:42:17.374 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:17.374 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:42:17.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:42:17.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:42:17.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:17.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822668 2020-03-11T22:42:17.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:17.717 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:17.717 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822668 2020-03-11T22:42:17.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:17.718 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:42:17.718 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2020-03-11T22:42:17.729 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:17.729 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 154618822668 2020-03-11T22:42:18.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822668 -lt 154618822668 2020-03-11T22:42:18.058 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:18.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:18.059 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:18.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:18.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:18.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:18.830 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:18.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:18.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:19.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:19.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:42:19.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:42:19.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:42:19.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:19.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:42:19.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:19.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:19.371 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:19.371 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:19.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:19.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:19.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:19.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:19.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:19.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:19.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:19.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:20.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:20.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:20.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:20.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:20.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:20.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:20.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:42:20.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:20.594 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:42:20.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:20.794 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:20.795 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:21.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:21.130 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:21.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:21.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:21.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:21.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:21.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:21.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:22.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:22.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:22.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:42:22.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:22.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:42:22.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:22.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:22.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:22.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:22.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:22.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:22.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:23.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:23.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:23.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:23.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:23.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:23.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:23.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:23.207 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:23.207 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:23.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:23.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:23.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:42:23.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:23.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:42:24.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:24.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:24.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:24.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:24.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:24.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:24.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:24.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:24.790 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:24.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:24.791 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:24.791 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:25.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:25.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:25.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:25.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:25.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:25.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:25.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:42:25.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:25.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:42:27.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:27.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:27.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:27.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:27.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:27.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:27.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:27.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:27.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:27.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:27.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:27.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:28.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:28.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:28.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:28.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:28.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:28.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:42:28.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:28.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:42:31.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:31.755 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:31.757 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:31.757 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:31.757 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:32.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:32.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:32.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:32.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:32.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:32.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:32.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:32.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:32.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:32.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:32.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:32.542 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:32.542 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:32.979 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:32.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:32.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:42:32.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:32.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:42:39.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:39.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:39.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:39.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:39.724 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:39.725 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:39.725 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:40.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:40.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:40.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:40.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:40.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:40.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:40.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:40.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:40.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:40.612 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:40.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:40.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:42:40.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:40.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:42:47.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:47.914 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:47.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:47.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:47.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:47.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:47.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:47.916 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:48.257 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:48.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:48.258 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:48.258 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:48.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:48.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:48.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:48.700 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:49.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:49.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:49.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:42:49.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:42:49.619 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:42:49.619 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "b09154b0-6144-4fd9-8d95-50275240a02f", 2020-03-11T22:42:49.619 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:42:49.615824+0000", 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:42:49.620 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:42:49.621 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:42:49.621 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:49.621 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:49.621 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.622 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:49.623 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.624 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:42:49.624 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.624 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.624 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:49.624 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 64 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:49.625 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.626 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.626 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.626 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.626 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.626 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.627 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.627 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.2 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.627 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.627 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.627 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.3 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.628 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.629 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.629 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.4 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.629 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.629 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.5 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.6 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.630 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.7 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.8 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.631 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.9 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.a is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.b is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.c is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.633 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.d is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.e is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.634 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.f is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.10 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.635 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.11 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.12 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.636 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.13 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.637 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.637 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.14 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.637 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.638 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.15 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.638 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.638 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.638 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.16 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.638 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.639 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.639 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.17 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.639 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.639 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.639 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.18 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.19 is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1a is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.640 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1b is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1c is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.641 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1d is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1e is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.642 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.643 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.643 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.1f is stuck undersized for 61s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:49.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.645 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:49.646 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.647 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:40:39.273772Z", 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:40:39.273772Z", 2020-03-11T22:42:49.650 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:42:49.651 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:42:49.652 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.653 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.654 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.654 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:42:49.654 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:42:49.654 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.654 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.655 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:42:49.656 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 42, 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:40:40.008033+0000", 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:41:46.344662+0000", 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:41:39.502268+0000", 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:41:40.918578+0000", 2020-03-11T22:42:49.657 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:42:49.658 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 7, 2020-03-11T22:42:49.659 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:42:49.660 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:42:49.661 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:41:01.865776+0000", 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 1, 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool", 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:42:49.662 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 0, 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 32, 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 32, 2020-03-11T22:42:49.663 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 32, 2020-03-11T22:42:49.664 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 32, 2020-03-11T22:42:49.664 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:42:49.664 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:42:49.664 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:42:49.664 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:42:49.665 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:42:49.665 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:42:49.665 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:42:49.665 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.665 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "41", 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "41", 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 0, 2020-03-11T22:42:49.666 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 0, 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [], 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:42:49.667 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:42:49.668 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "", 2020-03-11T22:42:49.669 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:42:49.670 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:42:49.670 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.670 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:42:49.670 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:42:49.670 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:42:49.671 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 0, 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.672 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "b7cca827-16ff-4908-ab2c-ccbfc3263ec2", 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:49.673 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:49.674 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:49.674 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:42:49.674 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 29, 2020-03-11T22:42:49.674 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 35, 2020-03-11T22:42:49.674 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 41, 2020-03-11T22:42:49.675 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:42:49.675 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:49.676 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:49.676 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.680 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.681 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:42:49.682 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.682 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.682 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.682 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.682 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.683 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.684 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39222 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/39222", 2020-03-11T22:42:49.685 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/39222", 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39222", 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39222", 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.686 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.687 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.687 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:49.687 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "aee7f2fc-043b-4c33-a93a-605dc6e31b60", 2020-03-11T22:42:49.687 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:42:49.687 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:49.688 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:49.688 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:49.688 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:42:49.688 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:42:49.688 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:42:49.689 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 22, 2020-03-11T22:42:49.689 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:42:49.689 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:49.689 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:49.689 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.690 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.690 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.690 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:49.691 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.693 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.693 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.693 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.693 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.695 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.695 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:49.695 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.695 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.695 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.696 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:49.697 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.697 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.697 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.697 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.697 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:49.698 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.699 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 34868 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.700 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/34868", 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/34868", 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34868", 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34868", 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.701 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "95b460b2-26b5-4e83-a391-67730479ac69", 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:42:49.702 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 29, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 36, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:42:49.703 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 32, 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:49.704 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.705 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.706 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:49.707 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.708 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.709 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.710 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.711 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:42:49.712 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:49.712 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 39418 2020-03-11T22:42:49.712 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.712 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.712 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/39418", 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/39418", 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39418", 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39418", 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:42:49.713 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:42:49.714 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:33.264233+0000", 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:46.983698+0000", 2020-03-11T22:42:49.715 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:49.716 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.716 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.716 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:49.716 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:33.758590+0000", 2020-03-11T22:42:49.716 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:53.341593+0000", 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:49.717 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:41:34.296860+0000", 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:42:49.718 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:59.579706+0000", 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:42:49.719 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:42:49.720 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:42:49.721 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:42:49.722 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/39222,v1:127.0.0.1:6805/39222]", 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:49.723 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:49.724 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:49.725 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:49.726 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:49.727 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:49.727 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/39222,v1:127.0.0.1:6803/39222]", 2020-03-11T22:42:49.728 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:49.728 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39222,v1:127.0.0.1:6809/39222]", 2020-03-11T22:42:49.728 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39222,v1:127.0.0.1:6807/39222]", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:49.729 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:49.730 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:49.730 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:49.730 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:49.730 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:42:49.730 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:49.731 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/34868,v1:127.0.0.1:6813/34868]", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:49.732 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:49.733 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:49.733 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:49.733 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:49.733 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:42:49.733 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:49.734 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:49.734 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:49.734 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:49.734 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:49.734 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:49.735 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/34868,v1:127.0.0.1:6811/34868]", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34868,v1:127.0.0.1:6817/34868]", 2020-03-11T22:42:49.736 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34868,v1:127.0.0.1:6815/34868]", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:49.737 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:49.738 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/39418,v1:127.0.0.1:6813/39418]", 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:42:49.739 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:49.740 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:49.741 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:42:49.741 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:49.741 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:49.741 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:49.741 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:42:49.742 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/39418,v1:127.0.0.1:6811/39418]", 2020-03-11T22:42:49.743 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39418,v1:127.0.0.1:6817/39418]", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39418,v1:127.0.0.1:6815/39418]", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:49.744 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:42:49.745 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 42, 2020-03-11T22:42:49.746 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:49.747 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.749 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:49.749 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:42:49.749 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:42:49.749 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.749 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.750 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:42:49.751 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.752 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:42:49.753 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.754 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:42:49.755 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.756 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:42:49.757 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.758 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.758 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:42:49.758 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.758 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:42:49.758 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:49.759 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.760 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:42:49.760 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.760 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:49.760 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.760 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:42:49.761 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.762 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.763 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.764 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:49.765 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:49.765 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:49.765 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.766 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:49.767 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.768 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:42:49.769 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:42:49.770 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.771 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.772 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.773 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:42:49.773 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.773 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:42:49.773 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:42:49.773 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:42:49.774 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:42:49.774 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:42:49.774 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:42:49.774 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:42:49.774 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.775 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:42:49.776 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:42:49.776 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:42:49.776 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.776 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.777 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:42:49.777 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:42:49.778 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 0, 2020-03-11T22:42:49.781 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:42:49.781 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:42:49.781 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:42:49.781 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.782 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:42:49.783 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:42:49.784 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.785 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:42:49.786 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:42:49.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:42:49.788 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 32, 2020-03-11T22:42:49.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 32, 2020-03-11T22:42:49.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:42:49.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 47947, 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:42:49.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:49.790 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:49.791 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:42:49.791 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:42:49.791 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:42:49.791 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:42:49.791 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:49.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:49.793 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:42:49.793 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:49.793 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:49.793 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:42:49.793 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:49.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:49.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:42:49.796 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:42:49.797 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:42:49.798 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.798 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 513, 2020-03-11T22:42:49.798 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 513, 2020-03-11T22:42:49.798 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 64, 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 64, 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:42:49.799 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:42:49.800 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 65, 2020-03-11T22:42:49.800 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:42:49.800 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:49.800 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:49.800 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179968, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33920, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 13, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145714, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311392832, 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:42:49.801 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866259968, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34734080, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 422423, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:49.802 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 13558, 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221211914 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:42:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:42:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:42:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:42:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:42:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 65, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179968, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33920, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 13, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145714, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311392832, 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:42:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:42:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866259968, 2020-03-11T22:42:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34734080, 2020-03-11T22:42:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 422423, 2020-03-11T22:42:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 13558, 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221211914 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:42:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:49.811 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:49.811 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:49.811 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:42:49.811 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:42:49.811 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:49.812 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.813 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 32, 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 47947, 2020-03-11T22:42:49.814 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 100, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 0, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 300, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:49.815 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:42:49.816 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:49.817 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:42:49.818 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:42:49.820 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:42:49.821 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19529728, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 142522, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:42:49.822 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 513, 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 513, 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 64, 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 64, 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:42:49.823 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 150323855379 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.824 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672974 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:49.825 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 154618822674 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:42:49.826 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 32 2020-03-11T22:42:49.827 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.827 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.827 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:42:49.827 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.828 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:42:49.828 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:42:49.829 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 32, 2020-03-11T22:42:49.829 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:49.829 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.829 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.830 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:42:49.830 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:42:49.830 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:42:49.830 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:49.831 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.831 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.831 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:42:49.831 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 32, 2020-03-11T22:42:49.831 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 32, 2020-03-11T22:42:49.832 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:49.832 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.832 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.833 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:42:49.833 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:42:49.833 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:42:49.833 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:42:49.834 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.834 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:42:49.834 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:42:49.834 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:42:49.835 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:40:40.008325+0000", 2020-03-11T22:42:49.835 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:42:49.835 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:42:49.835 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:42:49.836 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:42:49.836 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 383, 2020-03-11T22:42:49.836 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:42:49.837 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:42:49.837 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:42:49.837 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:42:49.839 INFO:tasks.workunit.client.0.smithi118.stderr:report 2032566390 2020-03-11T22:42:49.839 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:42:49.839 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:42:49.840 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:49.840 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery 2020-03-11T22:42:49.960 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:49.960 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.960 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.960 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.960 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.961 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.962 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.963 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.964 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.965 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.965 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 42'101 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.965 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:49.965 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:49.966 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:49.966 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:49.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:42:49.971 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:42:49.971 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:42:49.971 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:42:49.971 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:49.972 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:49.972 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:42:49.972 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:42:49.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:49.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:42:50.063 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:50.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:42:50.070 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:42:50.071 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:42:50.072 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:50.072 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:42:50.163 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:50.170 INFO:tasks.workunit.client.0.smithi118.stdout:revive divergent 1 2020-03-11T22:42:50.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:42:50.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:50.489 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:50.489 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.489 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.489 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.489 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.490 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.491 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.492 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.493 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 42'101 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.494 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 42:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:50.494 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:50.494 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:50.494 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:50.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:42:51.145 INFO:tasks.workunit.client.0.smithi118.stderr:noup is set 2020-03-11T22:42:51.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:42:51.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:51.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:51.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:42:51.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:51.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:51.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:51.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:51.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:51.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:51.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:51.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:51.159 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:42:51.160 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:51.161 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:42:51.162 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:42:51.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:42:51.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:51.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:51.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:51.182 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:42:51.179+0000 7f6da9d92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:51.184 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:42:51.180+0000 7f6da9d92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:51.185 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:42:51.182+0000 7f6da9d92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:51.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:42:51.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:42:51.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:42:51.698+0000 7f6da9d92ec0 -1 Falling back to public interface 2020-03-11T22:42:52.576 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:42:52.573+0000 7f6da9d92ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:42:56.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:42:56.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:56.571 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery divergent 2020-03-11T22:42:56.898 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:56.899 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.899 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.899 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.899 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.899 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.900 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.900 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.900 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.900 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.900 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.901 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 42:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.902 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.903 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 42'101 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:56.904 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:56.905 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:56.909 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:42:56.910 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:56.910 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:56.910 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:56.910 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:56.910 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:42:56.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:42:56.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:56.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:42:57.002 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:57.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:42:57.864 INFO:tasks.workunit.client.0.smithi118.stderr:noup is unset 2020-03-11T22:42:57.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:42:57.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:57.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:57.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:57.875 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:42:58.310 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 35 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/39222,v1:127.0.0.1:6803/39222] [v2:127.0.0.1:6804/39222,v1:127.0.0.1:6805/39222] exists,up b7cca827-16ff-4908-ab2c-ccbfc3263ec2 2020-03-11T22:42:58.311 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:42:58.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:58.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:58.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:58.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:58.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:58.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:58.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:58.652 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41950,v1:127.0.0.1:6819/41950] [v2:127.0.0.1:6820/41950,v1:127.0.0.1:6821/41950] exists,up aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:42:58.653 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:42:58.653 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:58.653 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:58.653 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:58.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:58.661 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:58.661 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:58.994 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/39418,v1:127.0.0.1:6811/39418] [v2:127.0.0.1:6812/39418,v1:127.0.0.1:6813/39418] exists,up 95b460b2-26b5-4e83-a391-67730479ac69 2020-03-11T22:42:58.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:58.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:58.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:58.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:59.315 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:59.316 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.316 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.316 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.316 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.317 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.317 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.317 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.317 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.318 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.318 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.318 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.318 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.318 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.319 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.319 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.319 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.319 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.320 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.320 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.320 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.320 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.321 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.321 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.321 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.321 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.322 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.322 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.322 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.322 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.323 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.323 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 42'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.323 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.323 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:59.323 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:59.324 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:59.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:42:59.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:59.326 INFO:tasks.workunit.client.0.smithi118.stdout:wait for peering 2020-03-11T22:42:59.645 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:59.645 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.616055+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613518+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615715+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614106+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614151+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611424+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611397+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.646 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614743+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613587+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613419+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613615+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611938+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614226+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613094+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.647 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612941+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614359+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612546+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613651+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612110+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612458+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:57.283435+0000 29'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.648 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614858+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.615589+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611389+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611556+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611477+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.614214+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611976+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.649 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.611906+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.650 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.612613+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.650 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:46.611958+0000 42'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.650 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:46.613035+0000 29'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:42:59.650 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:42:59.650 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:59.651 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:42:59.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:43:06.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:43:06.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:06.663 INFO:tasks.workunit.client.0.smithi118.stdout:killing divergent 1 2020-03-11T22:43:06.993 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:06.993 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.049589+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.993 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.051071+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.046957+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.047388+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.050095+0000 29'100 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.043892+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.041551+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.994 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.046568+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.047047+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.049663+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.044418+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.043612+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.046210+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.046755+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.995 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.049199+0000 29'100 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.048211+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044515+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.043986+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044043+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.045474+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.996 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.051224+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:42:58.048448+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.048826+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.044619+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.043778+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:42:58.050468+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.997 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.051517+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044162+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:42:58.045187+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:42:58.044319+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.043671+0000 42'101 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.050858+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:06.998 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:43:06.999 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:06.999 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:43:07.002 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:43:07.003 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:07.004 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:07.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:07.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:07.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:07.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:07.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17426.out 2020-03-11T22:43:07.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17426.out 2020-03-11T22:43:07.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:07.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:07.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:07.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:07.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:07.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17426.out 2020-03-11T22:43:07.679 INFO:tasks.workunit.client.0.smithi118.stderr:Exporting 1.0 info 1.0( v 29'100 (29'90,29'100] local-lis/les=46/47 n=3 ec=22/22 lis/c=46/46 les/c/f=47/47/0 sis=46 pruub=8.554128124s) 2020-03-11T22:43:07.692 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:43:07.708 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:43:07.729 INFO:tasks.workunit.client.0.smithi118.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:43:07.733 INFO:tasks.workunit.client.0.smithi118.stderr:Export successful 2020-03-11T22:43:07.734 INFO:tasks.workunit.client.0.smithi118.stdout: marking collection for removal 2020-03-11T22:43:07.734 INFO:tasks.workunit.client.0.smithi118.stdout:setting '_remove' omap key 2020-03-11T22:43:07.734 INFO:tasks.workunit.client.0.smithi118.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:43:07.783 INFO:tasks.workunit.client.0.smithi118.stderr:Remove successful 2020-03-11T22:43:08.250 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17426.out 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:08.251 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17426.out 2020-03-11T22:43:08.707 INFO:tasks.workunit.client.0.smithi118.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={22=1,41=32}} deleted_pools ) 2020-03-11T22:43:08.708 INFO:tasks.workunit.client.0.smithi118.stdout:Importing pgid 1.0 2020-03-11T22:43:08.708 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:43:08.708 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:43:08.792 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:43:08.792 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:43:08.800 INFO:tasks.workunit.client.0.smithi118.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:43:08.800 INFO:tasks.workunit.client.0.smithi118.stdout:snapset 0=[]:{} 2020-03-11T22:43:08.808 INFO:tasks.workunit.client.0.smithi118.stdout:write_pg epoch 47 info 1.0( v 29'100 (29'90,29'100] local-lis/les=46/47 n=3 ec=22/22 lis/c=46/46 les/c/f=47/47/0 sis=46 pruub=8.554128124s) 2020-03-11T22:43:08.816 INFO:tasks.workunit.client.0.smithi118.stdout:Import successful 2020-03-11T22:43:09.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:43:09.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:09.335 INFO:tasks.workunit.client.0.smithi118.stdout:reviving divergent 1 2020-03-11T22:43:09.661 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:09.661 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.049589+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.661 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.051071+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.046957+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.047388+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.050095+0000 29'100 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.043892+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.041551+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.046568+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.047047+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.049663+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.044418+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.043612+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.663 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.046210+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.046755+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.049199+0000 29'100 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.048211+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044515+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.043986+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044043+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.045474+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.051224+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:42:58.048448+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.048826+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.044619+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.043778+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.665 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:42:58.050468+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.051517+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:58.044162+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:42:58.045187+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:42:58.044319+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:42:58.043671+0000 42'101 47:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:58.050858+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:09.666 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:43:09.667 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:09.667 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:43:09.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:43:09.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:09.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:09.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:09.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:09.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:09.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:09.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:09.678 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:09.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:09.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:09.681 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:43:09.681 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:09.681 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:43:09.681 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:43:09.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:09.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:09.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:09.682 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:43:09.700 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:09.698+0000 7f5fa7abcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:09.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:09.699+0000 7f5fa7abcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:09.703 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:09.701+0000 7f5fa7abcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:10.020 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:43:10.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:10.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:10.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:10.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:10.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:10.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:10.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:10.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:10.022 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:10.219 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:10.216+0000 7f5fa7abcec0 -1 Falling back to public interface 2020-03-11T22:43:10.389 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41950,v1:127.0.0.1:6819/41950] [v2:127.0.0.1:6820/41950,v1:127.0.0.1:6821/41950] exists,up aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:43:10.389 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:10.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:10.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:10.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:10.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:10.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:10.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:10.391 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:10.746 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41950,v1:127.0.0.1:6819/41950] [v2:127.0.0.1:6820/41950,v1:127.0.0.1:6821/41950] exists,up aee7f2fc-043b-4c33-a93a-605dc6e31b60 2020-03-11T22:43:10.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:10.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:10.746 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:10.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:43:11.018 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:11.016+0000 7f5fa7abcec0 -1 osd.1 47 log_to_monitors {default=true} 2020-03-11T22:43:30.748 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:43:30.748 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:43:30.748 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:43:30.749 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:30.749 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:30.749 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:30.749 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:43:30.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:30.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:43:30.844 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:43:30.845 INFO:tasks.workunit.client.0.smithi118.stdout: "ops": [], 2020-03-11T22:43:30.845 INFO:tasks.workunit.client.0.smithi118.stdout: "num_ops": 0 2020-03-11T22:43:30.845 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:43:30.852 INFO:tasks.workunit.client.0.smithi118.stdout:allowing recovery 2020-03-11T22:43:30.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:43:30.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.636645+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:20.450730+0000 29'100 55:149 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:20.450684+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533342+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:11.849739+0000 29'100 57:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:20.637740+0000 29'100 57:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.612341+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:16.058220+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.535052+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:14.121237+0000 29'100 2020-03-11T22:43:14.121237+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.849307+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.637591+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.660466+0000 29'100 55:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:24.341193+0000 29'100 60:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:24.341131+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.176 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.849921+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:20.636898+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:43:25.585844+0000 29'100 59:153 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:25.585783+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533612+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.850406+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:43:29.327896+0000 29'100 61:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:29.327850+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:11.849822+0000 29'100 57:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.177 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:20.637321+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:18.660170+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:18.040578+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.851476+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.533744+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:43:18.659847+0000 47'101 55:142 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:20.637487+0000 29'100 57:152 [1,2,0] 1 [1,2,0] 1 29'100 2020-03-11T22:43:18.193847+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:11.846746+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.178 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:43:28.385652+0000 29'100 61:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:28.385577+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:18.659746+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:20.610683+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:43:11.849741+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533122+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:43:18.659836+0000 42'101 55:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.636476+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.179 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:43:31.180 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:31.180 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:43:31.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:31.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:43:31.305 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:31.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:31.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:43:31.433 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:31.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:31.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:43:31.561 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:31.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:43:31.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:31.569 INFO:tasks.workunit.client.0.smithi118.stdout:reading divergent objects 2020-03-11T22:43:31.888 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:31.888 INFO:tasks.workunit.client.0.smithi118.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.636645+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:20.450730+0000 29'100 55:149 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:20.450684+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533342+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:11.849739+0000 29'100 57:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:20.637740+0000 29'100 57:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.612341+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:16.058220+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.889 INFO:tasks.workunit.client.0.smithi118.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.535052+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:14.121237+0000 29'100 2020-03-11T22:43:14.121237+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.849307+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.637591+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.660466+0000 29'100 55:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:24.341193+0000 29'100 60:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:24.341131+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.849921+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:20.636898+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.890 INFO:tasks.workunit.client.0.smithi118.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:43:25.585844+0000 29'100 59:153 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:25.585783+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533612+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.850406+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:43:29.327896+0000 29'100 61:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:29.327850+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:11.849822+0000 29'100 57:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:20.637321+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:18.660170+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:18.040578+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.891 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:11.851476+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.533744+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:43:18.659847+0000 47'101 55:142 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:20.637487+0000 29'100 57:152 [1,2,0] 1 [1,2,0] 1 29'100 2020-03-11T22:43:18.193847+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:11.846746+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:43:28.385652+0000 29'100 61:134 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:28.385577+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.892 INFO:tasks.workunit.client.0.smithi118.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:18.659746+0000 29'100 59:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:20.610683+0000 29'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:43:11.849741+0000 29'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.533122+0000 29'100 60:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:43:18.659836+0000 42'101 55:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:20.636476+0000 29'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:41:01.955240+0000 0'0 2020-03-11T22:41:01.955240+0000 0 2020-03-11T22:43:31.893 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:43:31.894 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:31.894 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:43:31.901 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:43:31.902 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:43:31.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:31.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:43:31.951 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:31.951 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:43:31.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:31.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:43:32.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:32.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:43:32.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:43:32.043 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:32.043 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:43:32.070 INFO:tasks.workunit.client.0.smithi118.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:43:32.071 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:32.071 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:43:32.118 INFO:tasks.workunit.client.0.smithi118.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:43:32.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:43:32.120 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:32.121 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:32.129 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'103 (29'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2020-03-11T22:41:22.560625+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.129 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 20 _merge_object_divergent_entries: keeping 29'103 (29'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2020-03-11T22:41:22.560625+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.129 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:32.129 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'103 (29'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2020-03-11T22:41:22.560625+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.734+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'89 2020-03-11T22:43:32.130 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:41:22.561362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:41:22.561362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2020-03-11T22:43:32.131 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:41:22.561362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.736+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'84 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:41:22.561206+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:41:22.561206+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:43:32.132 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:41:22.561206+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2020-03-11T22:43:32.133 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-03-11T22:41:22.543204+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.134 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 20 _merge_object_divergent_entries: keeping 29'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-03-11T22:41:22.543204+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.134 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:32.134 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'102 last_divergent_update: 29'102 2020-03-11T22:43:32.134 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:43:32.136 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.737+0000 7f6d86f4b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:41:22.563439+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:41:22.563439+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: more recent entry found: 42'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4648.0:1 2020-03-11T22:42:14.906562+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:43:32.137 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4526.0:1 2020-03-11T22:41:22.543338+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.138 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4526.0:1 2020-03-11T22:41:22.543338+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:32.138 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:32.138 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:43:32.138 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:43:32.138 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:57.836+0000 7f6d85f49700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:32.139 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:32.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:32.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:32.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:32.154 INFO:tasks.workunit.client.0.smithi118.stdout:success 2020-03-11T22:43:32.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:43:32.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17426.out 2020-03-11T22:43:32.155 INFO:tasks.workunit.client.0.smithi118.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17426.out': No such file or directory 2020-03-11T22:43:32.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:43:32.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:43:32.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:43:32.577 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' does not exist 2020-03-11T22:43:32.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:43:32.585 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:32.585 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:32.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:32.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:32.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:32.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:32.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:43:32.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:32.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:32.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:32.694 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:32.694 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:32.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:32.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:32.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:32.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:32.698 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:32.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:32.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:32.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:32.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:32.700 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:32.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:32.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:32.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:32.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:32.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:32.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:32.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:32.722 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:32.722 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:43:32.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:32.724 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:32.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:32.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:32.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:32.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:32.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:32.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:32.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:32.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:32.728 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:32.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:32.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:32.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:32.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:32.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:32.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:32.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:43:32.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:32.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:32.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:43:32.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:43:32.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:32.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:32.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17426 2020-03-11T22:43:32.735 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:43:32.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:43:32.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:43:32.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:43:32.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:43:32.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:43:32.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:43:32.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:43:32.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:43:32.737 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:43:32.737 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:43:32.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:43:32.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:43:32.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:43:32.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:43:33.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:43:33.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:33.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:33.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:33.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:33.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:33.075 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:33.075 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:43:33.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:33.219 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:43:33.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:33.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get fsid 2020-03-11T22:43:33.305 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:33.306 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:33.307 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:33.307 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:33.307 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:33.307 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-mon.a.asok 2020-03-11T22:43:33.307 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:33.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17426/ceph-mon.a.asok config get mon_host 2020-03-11T22:43:33.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:43:33.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:43:33.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:43:33.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:43:33.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:43:33.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:43:33.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:43:34.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:43:34.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:34.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:34.285 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:34.314 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:34.314 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:43:34.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:34.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:34.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:43:34.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:34.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:34.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:34.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:34.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:34.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:34.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:34.318 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:34.318 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:34.318 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:34.318 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:34.318 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:34.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:34.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:34.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:34.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:34.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:34.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:34.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:34.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:34.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:34.320 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:43:34.321 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 89e499ae-fb1d-4d02-a7e0-88539fbd14b9' 2020-03-11T22:43:34.322 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:34.322 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:43:34.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCWaWleWemcExAAsyUHURUr2MJBWcnzOdDy9A== 2020-03-11T22:43:34.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCWaWleWemcExAAsyUHURUr2MJBWcnzOdDy9A=="}' 2020-03-11T22:43:34.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 -i td/divergent-priors/0/new.json 2020-03-11T22:43:34.682 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:34.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:43:34.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCWaWleWemcExAAsyUHURUr2MJBWcnzOdDy9A== --osd-uuid 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:43:34.720 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.716+0000 7fb717fb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:34.722 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.718+0000 7fb717fb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:34.723 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.719+0000 7fb717fb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:34.724 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.720+0000 7fb717fb7ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:34.724 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.720+0000 7fb717fb7ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:34.724 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.720+0000 7fb717fb7ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:34.757 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:34.754+0000 7fb717fb7ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:43:37.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:43:37.759 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:37.759 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:43:37.759 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:37.760 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:43:38.208 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:43:38.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:43:38.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:38.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:38.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:38.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:38.235 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:38.232+0000 7faaa2d30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:38.243 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:38.240+0000 7faaa2d30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:38.244 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:38.242+0000 7faaa2d30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:38.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:43:38.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:38.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:38.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:38.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:38.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:38.760 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:38.757+0000 7faaa2d30ec0 -1 Falling back to public interface 2020-03-11T22:43:38.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:39.500 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:39.497+0000 7faaa2d30ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:43:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:39.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:39.905 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43441,v1:127.0.0.1:6803/43441] [v2:127.0.0.1:6804/43441,v1:127.0.0.1:6805/43441] exists,up 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:40.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:40.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:40.267 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:40.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:40.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:40.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:40.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:40.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:40.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:43:40.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 bdbe9782-9f6f-479f-8860-f1efe9d4e8aa' 2020-03-11T22:43:40.271 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:40.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCcaWleqyStEBAAoX8KBUGoiL+dfjHd8uOS1w== 2020-03-11T22:43:40.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCcaWleqyStEBAAoX8KBUGoiL+dfjHd8uOS1w=="}' 2020-03-11T22:43:40.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bdbe9782-9f6f-479f-8860-f1efe9d4e8aa -i td/divergent-priors/1/new.json 2020-03-11T22:43:40.815 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:43:40.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:43:40.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcaWleqyStEBAAoX8KBUGoiL+dfjHd8uOS1w== --osd-uuid bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:43:40.855 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.852+0000 7f73ffcf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:40.856 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.854+0000 7f73ffcf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:40.858 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.856+0000 7f73ffcf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:40.859 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.856+0000 7f73ffcf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:40.859 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.856+0000 7f73ffcf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:40.859 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.856+0000 7f73ffcf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:40.990 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:40.987+0000 7f73ffcf6ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:43:43.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:43:43.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:43.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:43:43.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:43.993 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:43:44.444 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:43:44.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:44.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:44.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:44.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:44.445 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:43:44.473 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:44.468+0000 7ff186390ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:44.473 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:44.470+0000 7ff186390ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:44.474 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:44.472+0000 7ff186390ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:44.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:44.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:44.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:44.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:44.799 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:44.989 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:44.986+0000 7ff186390ec0 -1 Falling back to public interface 2020-03-11T22:43:45.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:45.832 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:45.829+0000 7ff186390ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:43:46.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:46.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:46.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:46.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:46.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:46.151 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:43:46.517 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43804,v1:127.0.0.1:6811/43804] [v2:127.0.0.1:6812/43804,v1:127.0.0.1:6813/43804] exists,up bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:43:46.517 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:43:46.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:46.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:46.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:46.520 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:43:46.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:46.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:43:46.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 b7258573-0afc-4577-85d0-d4c70cda4ce8' 2020-03-11T22:43:46.524 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:46.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCiaWle0nWvHxAALUQf5TUHRZO02g3pIaLBUg== 2020-03-11T22:43:46.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCiaWle0nWvHxAALUQf5TUHRZO02g3pIaLBUg=="}' 2020-03-11T22:43:46.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b7258573-0afc-4577-85d0-d4c70cda4ce8 -i td/divergent-priors/2/new.json 2020-03-11T22:43:47.057 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:43:47.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:43:47.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCiaWle0nWvHxAALUQf5TUHRZO02g3pIaLBUg== --osd-uuid b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:43:47.095 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.092+0000 7fd2f33edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.106 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.104+0000 7fd2f33edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.108 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.106+0000 7fd2f33edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.109 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.106+0000 7fd2f33edec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:47.109 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.106+0000 7fd2f33edec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:47.109 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.106+0000 7fd2f33edec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:47.173 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:47.170+0000 7fd2f33edec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:43:50.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:43:50.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:50.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:43:50.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:50.272 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:43:50.726 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:50.752 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:50.750+0000 7f6d64359ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:50.754 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:50.752+0000 7f6d64359ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:50.756 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:50.754+0000 7f6d64359ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:51.072 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:43:51.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:51.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:51.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:51.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:51.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:51.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:51.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:51.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:51.074 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:43:51.275 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:51.272+0000 7f6d64359ec0 -1 Falling back to public interface 2020-03-11T22:43:51.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:52.042 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:43:52.039+0000 7f6d64359ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:43:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:52.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:52.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:52.418 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:43:52.768 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44159,v1:127.0.0.1:6819/44159] [v2:127.0.0.1:6820/44159,v1:127.0.0.1:6821/44159] exists,up b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:43:52.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:52.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:52.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:52.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:43:53.427 INFO:tasks.workunit.client.0.smithi118.stderr:noout is set 2020-03-11T22:43:53.435 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:43:54.128 INFO:tasks.workunit.client.0.smithi118.stderr:noin is set 2020-03-11T22:43:54.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:43:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:nodown is set 2020-03-11T22:43:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:43:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:43:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:43:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:43:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:43:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:43:55.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:43:55.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:43:56.158 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' already exists 2020-03-11T22:43:56.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:43:57.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:43:57.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:43:57.760 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:43:57.770 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:57.770 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:57.770 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:57.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:57.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:57.862 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:58.234 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:58.235 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:43:58.235 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:43:58.235 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:58.235 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:58.235 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:58.385 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:43:58.385 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:43:58.385 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:58.385 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:58.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:43:58.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:43:58.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:58.526 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:58.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:43:58.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:43:58.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:58.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:43:58.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:58.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:58.668 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:43:58.668 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:58.669 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:43:58.669 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:43:58.673 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:43:58.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:58.997 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:43:58.997 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:59.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:59.998 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:00.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:44:00.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:00.350 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:44:00.350 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:00.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:00.351 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:44:00.352 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:00.352 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:44:00.352 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:44:00.353 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:00.353 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:44:00.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:44:00.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:00.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:44:00.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:00.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:00.712 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:44:00.712 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:00.713 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:44:00.713 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:44:00.713 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:00.713 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:44:01.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:44:01.081 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:01.082 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:01.082 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:01.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:01.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:01.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:01.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:44:01.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:01.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:01.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:02.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:44:02.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:44:02.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:44:02.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:44:02.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:44:02.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:02.298 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:02.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:02.645 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:44:02.645 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:44:02.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:02.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:02.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:02.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:44:02.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:44:02.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:02.774 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:02.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:44:02.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:44:02.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:02.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:03.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:44:03.033 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:44:03.033 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:03.033 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:44:03.034 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:03.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:03.034 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:44:03.035 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:03.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:44:03.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:44:03.036 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:44:03.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:44:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:04.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:04.379 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:04.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:44:04.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:04.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:44:04.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:04.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:04.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:44:04.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:04.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:44:04.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:44:04.734 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:44:04.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:05.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:44:05.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:05.240 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:44:05.241 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:05.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:05.242 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:44:05.242 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:05.242 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:44:05.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:44:05.243 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:44:05.243 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:05.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:44:05.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:05.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:05.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:05.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:05.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:05.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:44:05.610 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:05.610 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:05.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:05.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:05.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:05.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:05.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:05.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:05.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:05.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:06.044 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:44:06.045 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:44:06.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:06.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:06.045 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:06.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:44:06.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:44:06.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:06.174 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:06.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-03-11T22:44:06.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2020-03-11T22:44:06.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:06.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:06.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:44:06.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449' 2020-03-11T22:44:06.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:06.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:44:06.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:06.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:06.434 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:44:06.434 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:06.435 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:44:06.435 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:44:06.435 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:06.435 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:44:06.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836492 2020-03-11T22:44:06.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:07.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:07.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:08.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:44:08.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:08.129 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672970 2020-03-11T22:44:08.130 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:08.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:08.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672970 2020-03-11T22:44:08.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:08.132 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672970 2020-03-11T22:44:08.132 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-03-11T22:44:08.132 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:08.133 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672970 2020-03-11T22:44:08.477 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672970 2020-03-11T22:44:08.477 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:08.477 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:44:08.478 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:08.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:08.479 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:44:08.479 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:08.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:44:08.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:44:08.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:08.480 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:44:08.843 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509449 -lt 64424509449 2020-03-11T22:44:08.844 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:08.844 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:08.844 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:09.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:09.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:09.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:09.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:09.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:09.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:09.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:09.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:09.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:44:09.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:09.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:09.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:10.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:44:10.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:44:10.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:44:10.080 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:44:10.080 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:44:10.399 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi118.stdout:primary and soon to be divergent is 1 2020-03-11T22:44:10.726 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:10.726 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:43:56.563573+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:44:10.726 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:44:10.726 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:10.727 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:10.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:44:10.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:10.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:44:10.738 INFO:tasks.workunit.client.0.smithi118.stdout:writing initial objects 2020-03-11T22:44:10.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.836 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:44:10.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:44:10.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:44:10.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:44:11.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:44:11.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:44:11.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:44:11.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.241 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:44:11.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:44:11.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:44:11.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:44:11.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:44:11.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:44:11.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:44:11.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:44:11.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:44:11.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:44:11.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:44:12.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:44:12.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:44:12.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:44:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:44:12.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:44:12.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:44:12.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:44:12.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:44:12.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:44:12.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:44:12.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:44:12.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:44:12.824 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:44:12.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:12.867 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:44:13.023 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.023 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:44:13.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:44:13.108 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:44:13.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:44:13.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:44:13.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:44:13.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:44:13.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:44:13.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:44:13.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:44:13.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:44:13.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:44:13.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:44:13.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:44:13.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:13.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:44:14.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:44:14.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:44:14.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:44:14.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:44:14.233 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.234 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:44:14.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:44:14.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:44:14.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:44:14.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:44:14.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.613 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:44:14.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:44:14.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:44:14.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:44:14.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:44:14.910 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.910 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:44:14.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:14.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:44:15.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:44:15.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:44:15.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:44:15.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:44:15.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:44:15.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:44:15.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:44:15.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:44:15.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:44:15.737 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:44:15.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:44:15.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:44:15.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:15.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:44:16.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:44:16.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:44:16.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:44:16.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:44:16.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:44:16.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:44:16.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:44:16.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:44:16.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:44:16.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:44:16.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:44:16.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.785 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:44:16.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:44:16.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:44:16.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:16.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:44:17.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.027 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:44:17.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:44:17.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:44:17.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:44:17.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:44:17.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:17.352 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:44:17.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:17.394 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:17.395 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:17.395 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:17.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:17.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:17.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:17.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:17.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:17.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:17.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:17.452 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:17.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:17.816 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:44:17.817 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:44:17.817 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:17.817 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:17.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:44:17.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:44:17.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.944 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:18.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672975 2020-03-11T22:44:18.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2020-03-11T22:44:18.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:18.073 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:18.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509453 2020-03-11T22:44:18.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509453' 2020-03-11T22:44:18.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:18.203 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:44:18.203 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:18.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:18.204 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:44:18.204 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:18.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:44:18.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:44:18.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:18.206 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:44:18.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:44:18.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:19.576 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:19.577 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:19.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836496 2020-03-11T22:44:19.914 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:19.914 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672975 2020-03-11T22:44:19.914 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:19.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:19.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672975 2020-03-11T22:44:19.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:19.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672975 2020-03-11T22:44:19.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-03-11T22:44:19.917 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:19.917 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672975 2020-03-11T22:44:20.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 42949672975 2020-03-11T22:44:20.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:20.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509453 2020-03-11T22:44:20.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:20.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:20.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509453 2020-03-11T22:44:20.263 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:20.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509453 2020-03-11T22:44:20.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2020-03-11T22:44:20.264 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:20.264 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509453 2020-03-11T22:44:20.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509454 -lt 64424509453 2020-03-11T22:44:20.618 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:20.618 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:20.619 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:21.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:21.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:21.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:21.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:21.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:21.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:21.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:21.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:21.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:44:21.397 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:21.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:21.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:21.839 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:44:21.839 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:44:21.839 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:44:21.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:44:21.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:44:21.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:44:21.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:44:21.840 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:44:22.185 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:44:22.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:44:22.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:22.186 INFO:tasks.workunit.client.0.smithi118.stdout:blackholing osds 0 2 2020-03-11T22:44:22.504 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:22.505 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:43:56.563573+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:44:22.505 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:44:22.505 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:22.506 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:22.514 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:22.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:44:22.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:22.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:22.516 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:22.517 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:22.517 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:44:22.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:44:22.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:22.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:44:22.601 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:44:22.602 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:44:22.602 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:44:22.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:22.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:44:22.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:22.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:22.610 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:22.611 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:22.611 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:44:22.611 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:44:22.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:22.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:44:22.695 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:44:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:44:22.696 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:44:22.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:44:22.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:44:22.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:44:22.703 INFO:tasks.workunit.client.0.smithi118.stdout:writing divergent object 2020-03-11T22:44:23.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:44:23.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:44:23.819 INFO:tasks.workunit.client.0.smithi118.stdout:created pool test snap snap1 2020-03-11T22:44:23.821 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:44:23.821 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:44:23.822 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:44:33.822 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:44:33.826 INFO:tasks.workunit.client.0.smithi118.stderr:rados: no process found 2020-03-11T22:44:33.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:44:33.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:33.827 INFO:tasks.workunit.client.0.smithi118.stdout:killing all the osds 2020-03-11T22:44:34.156 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:34.156 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:43:56.563573+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:44:34.157 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:44:34.157 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:34.158 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:34.172 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:44:34.173 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:34.173 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:34.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:34.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:34.173 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:34.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:34.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:34.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:44:34.823 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already down. 2020-03-11T22:44:34.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:34.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:44:35.322 INFO:tasks.workunit.client.0.smithi118.stderr:osd.1 is already down. 2020-03-11T22:44:35.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:35.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:44:35.864 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already down. 2020-03-11T22:44:35.872 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:35.872 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:44:36.391 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already out. 2020-03-11T22:44:36.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:36.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:44:36.932 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already out. 2020-03-11T22:44:36.940 INFO:tasks.workunit.client.0.smithi118.stdout:bringing up non_divergent 0 2 2020-03-11T22:44:36.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:44:36.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:37.265 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:37.265 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:43:56.563573+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:44:37.265 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:44:37.265 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:37.266 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:37.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:37.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:37.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:37.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:37.281 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:37.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:37.283 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:44:37.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:37.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:37.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:37.284 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:37.284 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:44:37.303 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:37.300+0000 7f61032deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:37.310 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:37.302+0000 7f61032deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:37.320 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:37.304+0000 7f61032deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:37.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:37.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:37.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:37.625 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:44:37.825 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:37.823+0000 7f61032deec0 -1 Falling back to public interface 2020-03-11T22:44:37.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:38.791 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:38.788+0000 7f61032deec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:44:38.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:38.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:38.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:38.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:38.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:38.974 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:44:39.326 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48555,v1:127.0.0.1:6803/48555] [v2:127.0.0.1:6804/48555,v1:127.0.0.1:6805/48555] exists,up 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:44:39.326 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:44:39.327 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:39.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:39.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:44:39.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:39.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:44:39.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:44:39.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:39.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:39.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:39.352 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:39.349+0000 7fed49253ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.354 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:39.351+0000 7fed49253ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.356 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:39.353+0000 7fed49253ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:39.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:39.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:39.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:39.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:39.696 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:44:39.874 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:39.871+0000 7fed49253ec0 -1 Falling back to public interface 2020-03-11T22:44:40.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:40.742 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:44:40.740+0000 7fed49253ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:44:41.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:41.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:41.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:41.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:41.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:41.039 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:44:41.386 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48749,v1:127.0.0.1:6811/48749] [v2:127.0.0.1:6812/48749,v1:127.0.0.1:6813/48749] exists,up b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:44:41.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:41.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:41.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:41.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:41.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:44:41.934 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already in. 2020-03-11T22:44:41.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:41.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:44:42.500 INFO:tasks.workunit.client.0.smithi118.stderr:osd.2 is already in. 2020-03-11T22:44:42.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:44:47.510 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:44:47.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:44:47.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:44:47.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:47.511 INFO:tasks.workunit.client.0.smithi118.stdout:writing non-divergent object existing_4 2020-03-11T22:44:47.839 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:47.840 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:42.546853+0000 28'100 39:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:44:47.840 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:44:47.840 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:47.841 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:44:47.849 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:44:47.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:47.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:47.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:47.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:47.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:47.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:47.935 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:47.935 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:47.935 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:47.936 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:47.936 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:47.936 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:47.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:47.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:47.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:47.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:47.992 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:48.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:48.334 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:44:48.334 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:44:48.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:48.335 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:48.335 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:48.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855364 2020-03-11T22:44:48.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364' 2020-03-11T22:44:48.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:48.464 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:48.553 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:48.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:48.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364 1-' 2020-03-11T22:44:48.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:48.563 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:48.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822660 2020-03-11T22:44:48.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364 1- 2-154618822660' 2020-03-11T22:44:48.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:48.691 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855364 2020-03-11T22:44:48.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:48.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:48.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855364 2020-03-11T22:44:48.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:48.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855364 2020-03-11T22:44:48.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2020-03-11T22:44:48.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:48.699 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 150323855364 2020-03-11T22:44:49.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855362 -lt 150323855364 2020-03-11T22:44:49.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:50.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:50.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:50.385 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855365 -lt 150323855364 2020-03-11T22:44:50.386 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:50.386 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:50.386 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:50.388 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:50.388 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:50.389 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:50.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:50.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:50.390 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:50.390 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:44:50.730 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 2020-03-11T22:44:50.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672978: unary operator expected 2020-03-11T22:44:50.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:50.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822660 2020-03-11T22:44:50.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:50.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:50.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822660 2020-03-11T22:44:50.733 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:50.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822660 2020-03-11T22:44:50.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822660' 2020-03-11T22:44:50.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:50.734 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 154618822660 2020-03-11T22:44:51.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822660 2020-03-11T22:44:51.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:51.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:51.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:51.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:51.870 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:51.870 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:51.870 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:52.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:52.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:52.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:52.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:52.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:52.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:52.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:52.414 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:52.415 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:52.744 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:52.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:52.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:52.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:53.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:53.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:53.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:53.188 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:53.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:53.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:53.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:53.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:53.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:53.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:53.872 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:53.872 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:53.872 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:53.872 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:53.872 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:53.873 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:53.873 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:54.199 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:54.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:54.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:54.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:54.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:55.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:55.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:55.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:55.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:55.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:55.481 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:55.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:55.483 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:55.483 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:55.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:55.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:55.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:55.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:55.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:55.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:55.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:55.818 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:56.259 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:56.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:56.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:56.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:56.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:56.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:56.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:56.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:56.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:56.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:56.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:56.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:56.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:56.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:57.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:57.500 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:57.501 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:57.833 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:57.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:57.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:57.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:58.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:58.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:58.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:58.289 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:58.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:58.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:58.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:58.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:58.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:45:00.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:00.347 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:00.348 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:00.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:00.685 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:00.686 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:00.686 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:01.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:01.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:01.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:01.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:01.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:01.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:01.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:01.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:01.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:01.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:01.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:01.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:45:01.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:01.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:45:04.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:04.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:04.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:04.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:04.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:04.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:04.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:04.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:05.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:05.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:05.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:05.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:05.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:05.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:05.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:05.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:06.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:06.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:06.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:45:06.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:06.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:45:12.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:12.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:12.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:12.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:12.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:12.410 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:12.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:12.411 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:12.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:12.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:12.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:12.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:13.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:13.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:13.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:13.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:13.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:13.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:13.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:13.193 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:13.193 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:13.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:13.632 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:13.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:45:13.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:13.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:45:20.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:20.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:20.934 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:20.934 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:20.935 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:21.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:21.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:21.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:21.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:21.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:21.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:21.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:21.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:21.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:21.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:21.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:21.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:21.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:22.174 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:22.174 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:22.174 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:45:22.175 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:45:22.609 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "bf78e706-ff02-4a56-b5e5-35381554c109", 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:45:22.606081+0000", 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:45:22.610 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:45:22.611 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:45:22.612 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:45:22.613 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:45:22.614 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.615 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:45:22.616 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.617 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.619 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:43:32.756350Z", 2020-03-11T22:45:22.620 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:43:32.756350Z", 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:45:22.621 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.622 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.623 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:45:22.624 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:45:22.625 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.626 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:45:22.626 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 39, 2020-03-11T22:45:22.626 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:45:22.629 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:43:33.486765+0000", 2020-03-11T22:45:22.629 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:44:42.427693+0000", 2020-03-11T22:45:22.629 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:44:40.808627+0000", 2020-03-11T22:45:22.629 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:44:42.227055+0000", 2020-03-11T22:45:22.629 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:45:22.630 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 8, 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:45:22.631 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:45:22.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:43:55.806235+0000", 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 16389, 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 3, 2020-03-11T22:45:22.633 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 1, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 1, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:45:22.634 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 1, 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 1, 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 1, 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:45:22.635 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "29", 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:45:22.636 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "pool", 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 1, 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 29, 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [ 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.637 INFO:tasks.workunit.client.0.smithi118.stdout: "snapid": 1, 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: "stamp": "2020-03-11T22:44:23.730075+0000", 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "snap1" 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:45:22.638 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:45:22.639 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:45:22.640 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:45:22.640 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:45:22.640 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:45:22.640 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:45:22.640 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:45:22.641 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:45:22.641 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:45:22.641 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:45:22.641 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.642 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:45:22.642 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:45:22.642 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:45:22.642 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:45:22.643 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:45:22.643 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:45:22.643 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:45:22.643 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:45:22.644 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 8192, 2020-03-11T22:45:22.644 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:45:22.644 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:45:22.644 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:45:22.645 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:45:22.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.645 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.645 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:45:22.646 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.646 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:45:22.646 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "89e499ae-fb1d-4d02-a7e0-88539fbd14b9", 2020-03-11T22:45:22.647 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:45:22.647 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:45:22.647 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:45:22.648 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:45:22.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 5, 2020-03-11T22:45:22.648 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:45:22.648 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 35, 2020-03-11T22:45:22.649 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 38, 2020-03-11T22:45:22.649 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 30, 2020-03-11T22:45:22.650 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:45:22.650 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:45:22.650 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.650 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.650 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.651 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:45:22.651 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.651 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.651 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.652 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.652 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:45:22.652 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.652 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.653 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.653 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.653 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:45:22.653 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.654 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.654 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.654 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:45:22.655 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.655 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.655 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.655 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.655 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:45:22.656 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.656 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.656 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.656 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.657 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:22.657 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.657 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.657 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.658 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:45:22.658 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.658 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.659 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.659 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:45:22.659 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.660 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.660 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.660 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.660 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:22.660 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.661 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.661 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.661 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:45:22.661 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.662 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.662 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.662 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:45:22.662 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48555 2020-03-11T22:45:22.663 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.663 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.663 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.663 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/48555", 2020-03-11T22:45:22.664 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/48555", 2020-03-11T22:45:22.664 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48555", 2020-03-11T22:45:22.664 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48555", 2020-03-11T22:45:22.664 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:45:22.665 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:45:22.665 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:45:22.665 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.666 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.666 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.666 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:45:22.666 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "bdbe9782-9f6f-479f-8860-f1efe9d4e8aa", 2020-03-11T22:45:22.667 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:45:22.667 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:45:22.667 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:45:22.667 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:45:22.668 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:45:22.668 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:45:22.668 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:45:22.668 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 24, 2020-03-11T22:45:22.669 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:45:22.669 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:45:22.669 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:45:22.669 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.669 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.671 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.671 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:22.671 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.671 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.671 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.672 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:45:22.673 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.673 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.673 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.673 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.674 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.675 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.676 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.677 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.678 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/43804", 2020-03-11T22:45:22.679 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/43804", 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43804", 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43804", 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.680 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "b7258573-0afc-4577-85d0-d4c70cda4ce8", 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 15, 2020-03-11T22:45:22.681 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 28, 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 36, 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 32, 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.682 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:22.683 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.684 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.685 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:22.686 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.687 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:45:22.688 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:22.689 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 48749 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/48749", 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/48749", 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48749", 2020-03-11T22:45:22.690 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48749", 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:45:22.691 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:44:34.632238+0000", 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:45:22.692 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:40.533006+0000", 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:44:35.125934+0000", 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:45:22.693 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:45:22.694 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:45:22.694 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:45:22.694 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:46.818101+0000", 2020-03-11T22:45:22.694 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:44:35.635519+0000", 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:45:22.695 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:53.043693+0000", 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:45:22.696 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:45:22.697 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:45:22.698 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:45:22.698 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:45:22.698 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.698 INFO:tasks.workunit.client.0.smithi118.stdout: "myprofile": { 2020-03-11T22:45:22.698 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-device-class": "", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-failure-domain": "osd", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-root": "default", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "1", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:45:22.699 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van", 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: "w": "8" 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:45:22.700 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:45:22.701 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:45:22.701 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.701 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:45:22.701 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.701 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/48555,v1:127.0.0.1:6805/48555]", 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:22.702 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:22.703 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:22.704 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:45:22.705 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/48555,v1:127.0.0.1:6803/48555]", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48555,v1:127.0.0.1:6809/48555]", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48555,v1:127.0.0.1:6807/48555]", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:22.706 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:22.707 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/43804,v1:127.0.0.1:6813/43804]", 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:45:22.708 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:22.709 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:22.710 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:45:22.711 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/43804,v1:127.0.0.1:6811/43804]", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43804,v1:127.0.0.1:6817/43804]", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43804,v1:127.0.0.1:6815/43804]", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:45:22.712 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:22.713 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:45:22.714 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/48749,v1:127.0.0.1:6813/48749]", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:22.715 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:22.716 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:45:22.717 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/48749,v1:127.0.0.1:6811/48749]", 2020-03-11T22:45:22.718 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:45:22.719 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48749,v1:127.0.0.1:6817/48749]", 2020-03-11T22:45:22.719 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48749,v1:127.0.0.1:6815/48749]", 2020-03-11T22:45:22.719 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:45:22.719 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:22.720 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 39, 2020-03-11T22:45:22.721 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.722 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:45:22.723 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:45:22.724 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:45:22.725 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.726 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.728 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:45:22.729 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.730 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:45:22.731 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:45:22.732 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.733 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:45:22.734 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.735 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:45:22.736 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.737 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:45:22.738 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:45:22.739 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:45:22.740 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.741 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.742 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:45:22.743 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:45:22.744 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.745 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 1, 2020-03-11T22:45:22.746 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "test", 2020-03-11T22:45:22.747 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 1, 2020-03-11T22:45:22.747 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 3, 2020-03-11T22:45:22.747 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 3, 2020-03-11T22:45:22.747 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 3, 2020-03-11T22:45:22.747 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:45:22.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.748 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:45:22.748 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 5 2020-03-11T22:45:22.748 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "set_choose_tries", 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 100 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:45:22.749 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:45:22.750 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:45:22.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.750 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.750 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_indep", 2020-03-11T22:45:22.750 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:45:22.751 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:45:22.752 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:45:22.753 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:45:22.754 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:45:22.755 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:45:22.756 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.757 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:45:22.758 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 1, 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48430, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 101, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 1, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 303, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:45:22.762 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:22.763 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:45:22.764 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:22.765 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:22.766 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:45:22.767 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:45:22.767 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.767 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:45:22.767 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:45:22.767 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:22.768 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 18, 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:22.769 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 3, 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 3, 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:45:22.770 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:45:22.771 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179456, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33472, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 11, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145716, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393344, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:45:22.772 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866784256, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34275328, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1517397, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:45:22.773 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 11530, 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221213942 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:22.774 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:22.775 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:45:22.776 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:22.777 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:45:22.778 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3179456, 2020-03-11T22:45:22.778 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 33472, 2020-03-11T22:45:22.778 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 11, 2020-03-11T22:45:22.778 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145716, 2020-03-11T22:45:22.778 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311393344, 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318866784256, 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 34275328, 2020-03-11T22:45:22.779 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1517397, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 11530, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221213942 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.780 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:45:22.781 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.782 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:45:22.783 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48430, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 101, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 1, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 303, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:22.784 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:45:22.785 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:22.786 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:45:22.787 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:22.788 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.789 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19857408, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1241088, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:22.790 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 18, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 3, 2020-03-11T22:45:22.791 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 3, 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:45:22.792 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 150323855371 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672978 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:45:22.793 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 154618822667 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 1 2020-03-11T22:45:22.794 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:45:22.795 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2147483647, 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.796 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.797 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 1, 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:45:22.798 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:45:22.799 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:43:33.487085+0000", 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 329, 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:45:22.800 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:45:22.801 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:45:22.801 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr:report 3138451560 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:22.803 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:22.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:45:22.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:22.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:45:22.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:45:23.242 INFO:tasks.workunit.client.0.smithi118.stderr:PG '1.0' not found 2020-03-11T22:45:23.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:23.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:45:23.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:23.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:23.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:45:23.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:23.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:45:23.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:23.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:23.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:23.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:23.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:45:23.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:45:23.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:23.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:23.773 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:23.774 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:45:23.774 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:45:23.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:45:23.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:23.774 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:23.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:23.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:23.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:23.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:23.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:45:23.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:23.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:23.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:23.777 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:45:23.790 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:23.788+0000 7f93fec5dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:23.801 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:23.798+0000 7f93fec5dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:23.803 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:23.800+0000 7f93fec5dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:24.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:45:24.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:24.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:24.119 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:45:24.325 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:24.317+0000 7f93fec5dec0 -1 Falling back to public interface 2020-03-11T22:45:24.463 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48555,v1:127.0.0.1:6803/48555] [v2:127.0.0.1:6804/48555,v1:127.0.0.1:6805/48555] exists,up 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:45:24.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:24.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:24.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:24.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:24.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:24.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:24.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:45:24.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:24.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:24.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:45:24.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:24.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:45:24.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:45:25.337 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:25.334+0000 7f93fec5dec0 -1 osd.0 39 log_to_monitors {default=true} 2020-03-11T22:45:25.356 INFO:tasks.workunit.client.0.smithi118.stderr:PG '1.0' not found 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:45:25.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:45:25.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:25.878 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:25.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:45:25.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:45:25.881 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:45:25.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:25.882 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:45:25.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:45:25.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:25.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:25.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:25.906 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:25.903+0000 7f1f19b60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:25.907 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:25.905+0000 7f1f19b60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:25.910 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:25.907+0000 7f1f19b60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:26.235 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:26.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:26.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:26.237 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:45:26.440 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:26.437+0000 7f1f19b60ec0 -1 Falling back to public interface 2020-03-11T22:45:26.584 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48749,v1:127.0.0.1:6811/48749] [v2:127.0.0.1:6812/48749,v1:127.0.0.1:6813/48749] exists,up b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:45:26.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:45:26.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:26.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:45:26.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:45:27.357 INFO:tasks.workunit.client.0.smithi118.stderr:PG '1.0' not found 2020-03-11T22:45:27.443 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:45:27.441+0000 7f1f19b60ec0 -1 osd.2 39 log_to_monitors {default=true} 2020-03-11T22:45:27.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:27.875 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:45:27.876 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:27.876 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:27.876 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:27.877 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:27.877 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:27.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:27.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:27.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:27.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:27.947 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:28.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:45:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:2' 2020-03-11T22:45:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:28.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:28.437 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=176093659139 2020-03-11T22:45:28.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139' 2020-03-11T22:45:28.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:28.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:28.528 INFO:tasks.workunit.client.0.smithi118.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:45:28.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:45:28.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1-' 2020-03-11T22:45:28.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:28.537 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:28.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=188978561027 2020-03-11T22:45:28.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1- 2-188978561027' 2020-03-11T22:45:28.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:28.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-176093659139 2020-03-11T22:45:28.666 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:28.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:28.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-176093659139 2020-03-11T22:45:28.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:28.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=176093659139 2020-03-11T22:45:28.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 176093659139' 2020-03-11T22:45:28.668 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:28.669 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 176093659139 2020-03-11T22:45:29.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 176093659139 2020-03-11T22:45:29.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:30.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:30.011 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:30.361 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 176093659139 -lt 176093659139 2020-03-11T22:45:30.362 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:30.362 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:45:30.362 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:30.363 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:30.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:45:30.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:30.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:45:30.364 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:45:30.365 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:30.365 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 2020-03-11T22:45:30.707 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 2020-03-11T22:45:30.707 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672978: unary operator expected 2020-03-11T22:45:30.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:30.708 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-188978561027 2020-03-11T22:45:30.708 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:30.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:30.709 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-188978561027 2020-03-11T22:45:30.709 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:30.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=188978561027 2020-03-11T22:45:30.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 188978561027' 2020-03-11T22:45:30.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:30.711 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 188978561027 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 188978561027 -lt 188978561027 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:31.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:31.492 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:31.493 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:31.834 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:31.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:31.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:31.835 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:32.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:45:32.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:32.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:32.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:32.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:32.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:32.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:32.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:32.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:32.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:32.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:32.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:32.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:33.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:33.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:33.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:33.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:33.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:33.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:33.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:33.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:33.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:45:33.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:33.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:45:33.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:33.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:33.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:34.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:34.142 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:34.143 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:34.143 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:34.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:34.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:34.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:34.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:34.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:34.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:34.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:34.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:34.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:35.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:35.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:35.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:45:35.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:35.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:45:35.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:35.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:35.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:35.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:35.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:35.763 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:35.763 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:35.764 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:36.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:36.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:36.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:36.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:36.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:36.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:36.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:36.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:36.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:36.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:36.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:36.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:45:36.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:36.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:45:37.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:37.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:37.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:37.447 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:37.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:37.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:37.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:37.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:37.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:37.784 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:37.784 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:37.784 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:38.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:38.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:38.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:38.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:38.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:38.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:38.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:45:38.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:38.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:45:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:40.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:40.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:40.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:40.611 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:40.611 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:40.611 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:41.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:41.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:41.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:41.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:41.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:41.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:41.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:45:41.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:41.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:45:44.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:44.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:44.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:44.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:44.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:44.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:44.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:44.697 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:45.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:45.033 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:45.033 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:45.033 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:45.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:45.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:45.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:45.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:45.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:45.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:45.475 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:45.475 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:45.475 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:45.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:45.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:45.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:45:45.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:45.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:45:52.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:52.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:52.598 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:52.598 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:52.598 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:52.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:52.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:52.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:52.661 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:52.662 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:52.662 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:52.662 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:53.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:53.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:53.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:53.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:53.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:53.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:53.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:45:53.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:53.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:46:00.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:00.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:00.845 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:00.845 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:00.845 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:00.845 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:00.846 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:00.846 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:01.185 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:01.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:01.186 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:01.186 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:01.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:01.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:01.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:01.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:01.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:01.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:01.627 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:01.627 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:01.627 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:02.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:02.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:02.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:46:02.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:46:02.502 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_fingerprint": "bf78e706-ff02-4a56-b5e5-35381554c109", 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "timestamp": "2020-03-11T22:46:02.499515+0000", 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "tag": "", 2020-03-11T22:46:02.503 INFO:tasks.workunit.client.0.smithi118.stdout: "health": { 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "status": "HEALTH_WARN", 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "checks": { 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:46:02.504 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 17 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [], 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: "OSD_DOWN": { 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:46:02.505 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 osds down", 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "osd.1 (root=default,host=smithi118) is down" 2020-03-11T22:46:02.506 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: "PG_DEGRADED": { 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:46:02.507 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.508 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "summary": { 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:46:02.509 INFO:tasks.workunit.client.0.smithi118.stdout: "count": 1 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: "detail": [ 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:46:02.510 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: "muted": false 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: "mutes": [] 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.511 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_first_committed": 1, 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap_last_committed": 1, 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "monmap": { 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:43:32.756350Z", 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:43:32.756350Z", 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release": 15, 2020-03-11T22:46:02.512 INFO:tasks.workunit.client.0.smithi118.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "features": { 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "persistent": [ 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "kraken", 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "luminous", 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "mimic", 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap-prune", 2020-03-11T22:46:02.513 INFO:tasks.workunit.client.0.smithi118.stdout: "nautilus", 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: "octopus" 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: "optional": [] 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: "mons": [ 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.514 INFO:tasks.workunit.client.0.smithi118.stdout: "rank": 0, 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "a", 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 0 2020-03-11T22:46:02.515 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 0, 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 0 2020-03-11T22:46:02.516 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: "quorum": [ 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: 0 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap": { 2020-03-11T22:46:02.517 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 45, 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "fsid": "9298b8d8-9f52-4152-9823-0eb7fba47eea", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "created": "2020-03-11T22:43:33.486765+0000", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:45:27.989774+0000", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "last_up_change": "2020-03-11T22:45:27.689153+0000", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "last_in_change": "2020-03-11T22:44:42.227055+0000", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:46:02.518 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_num": 5800384, 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_set": [ 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "nodown", 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "noin", 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "noout", 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "pglog_hardlimit", 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snapdirs", 2020-03-11T22:46:02.519 INFO:tasks.workunit.client.0.smithi118.stdout: "recovery_deletes", 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "sortbitwise" 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_version": 8, 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:46:02.520 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_snapshot": "", 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_max": 1, 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "max_osd": 3, 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "min_compat_client": "jewel", 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "require_osd_release": "octopus", 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: "pools": [ 2020-03-11T22:46:02.521 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_name": "test", 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "create_time": "2020-03-11T22:43:55.806235+0000", 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "flags": 16389, 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 3, 2020-03-11T22:46:02.522 INFO:tasks.workunit.client.0.smithi118.stdout: "size": 3, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 2, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_rule": 1, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "object_hash": 2, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num": 1, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num": 1, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_placement_num_target": 1, 2020-03-11T22:46:02.523 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_target": 1, 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_num_pending": 1, 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pg_merge_meta": { 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "source_pgid": "0.0", 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "ready_epoch": 0, 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_started": 0, 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "last_epoch_clean": 0, 2020-03-11T22:46:02.524 INFO:tasks.workunit.client.0.smithi118.stdout: "source_version": "0'0", 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "target_version": "0'0" 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "last_change": "29", 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend": "0", 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:46:02.525 INFO:tasks.workunit.client.0.smithi118.stdout: "auid": 0, 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_mode": "pool", 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_seq": 1, 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_epoch": 29, 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_snaps": [ 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.526 INFO:tasks.workunit.client.0.smithi118.stdout: "snapid": 1, 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: "stamp": "2020-03-11T22:44:23.730075+0000", 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "snap1" 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps": "[]", 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_bytes": 0, 2020-03-11T22:46:02.527 INFO:tasks.workunit.client.0.smithi118.stdout: "quota_max_objects": 0, 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "tiers": [], 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "tier_of": -1, 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "read_tier": -1, 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "write_tier": -1, 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_mode": "none", 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_bytes": 0, 2020-03-11T22:46:02.528 INFO:tasks.workunit.client.0.smithi118.stdout: "target_max_objects": 0, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_flush_age": 0, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "cache_min_evict_age": 0, 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:46:02.529 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_params": { 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "none" 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_period": 0, 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_count": 0, 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "use_gmt_hitset": true, 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:46:02.530 INFO:tasks.workunit.client.0.smithi118.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "grade_table": [], 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "stripe_width": 8192, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "expected_num_objects": 0, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "fast_read": false, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "options": {}, 2020-03-11T22:46:02.531 INFO:tasks.workunit.client.0.smithi118.stdout: "application_metadata": {} 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: "osds": [ 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "89e499ae-fb1d-4d02-a7e0-88539fbd14b9", 2020-03-11T22:46:02.532 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 35, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 39, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 41, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 44, 2020-03-11T22:46:02.533 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 40, 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:46:02.534 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.537 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.538 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.539 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.540 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50198 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.541 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6803/50198", 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6805/50198", 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50198", 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50198", 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:46:02.542 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "bdbe9782-9f6f-479f-8860-f1efe9d4e8aa", 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 0, 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:46:02.543 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 0, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 0, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 10, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 24, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 31, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:46:02.544 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.545 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.546 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.547 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.548 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.549 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:46:02.550 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.551 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 43804 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/43804", 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/43804", 2020-03-11T22:46:02.552 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43804", 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43804", 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: "exists" 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.553 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "uuid": "b7258573-0afc-4577-85d0-d4c70cda4ce8", 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 1, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "in": 1, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 1, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_affinity": 1, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_begin": 36, 2020-03-11T22:46:02.554 INFO:tasks.workunit.client.0.smithi118.stdout: "last_clean_end": 39, 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 44, 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "up_thru": 0, 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "down_at": 43, 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "lost_at": 0, 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addrs": { 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.555 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:46:02.556 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addrs": { 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.557 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.558 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addrs": { 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.559 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addrs": { 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: "addrvec": [ 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v2", 2020-03-11T22:46:02.561 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "v1", 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: "nonce": 50430 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.562 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "public_addr": "127.0.0.1:6811/50430", 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "cluster_addr": "127.0.0.1:6813/50430", 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50430", 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50430", 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "state": [ 2020-03-11T22:46:02.563 INFO:tasks.workunit.client.0.smithi118.stdout: "exists", 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: "up" 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_xinfo": [ 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:46:02.564 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:45:25.434346+0000", 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:40.533006+0000", 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.565 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:44:35.125934+0000", 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:46.818101+0000", 2020-03-11T22:46:02.566 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "down_stamp": "2020-03-11T22:45:27.514448+0000", 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_probability": 0, 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "laggy_interval": 0, 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "features": 4540138292836696063, 2020-03-11T22:46:02.567 INFO:tasks.workunit.client.0.smithi118.stdout: "old_weight": 0, 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:53.043693+0000", 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: "dead_epoch": 0 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap": [], 2020-03-11T22:46:02.568 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_upmap_items": [], 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_temp": [], 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "primary_temp": [], 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "blacklist": {}, 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "erasure_code_profiles": { 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "default": { 2020-03-11T22:46:02.569 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "2", 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van" 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "myprofile": { 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-device-class": "", 2020-03-11T22:46:02.570 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-failure-domain": "osd", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "crush-root": "default", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "k": "2", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "m": "1", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "plugin": "jerasure", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "technique": "reed_sol_van", 2020-03-11T22:46:02.571 INFO:tasks.workunit.client.0.smithi118.stdout: "w": "8" 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: "removed_snaps_queue": [], 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: "new_removed_snaps": [], 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: "new_purged_snaps": [], 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: "crush_node_flags": {}, 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: "device_class_flags": {} 2020-03-11T22:46:02.572 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_metadata": [ 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6804/50198,v1:127.0.0.1:6805/50198]", 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:46:02.573 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:46:02.574 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6802/50198,v1:127.0.0.1:6803/50198]", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50198,v1:127.0.0.1:6809/50198]", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50198,v1:127.0.0.1:6807/50198]", 2020-03-11T22:46:02.577 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:46:02.578 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:46:02.579 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/43804,v1:127.0.0.1:6813/43804]", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:46:02.580 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:46:02.581 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:46:02.582 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/43804,v1:127.0.0.1:6811/43804]", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:46:02.583 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43804,v1:127.0.0.1:6817/43804]", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43804,v1:127.0.0.1:6815/43804]", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:46:02.584 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:46:02.585 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "arch": "x86_64", 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "back_addr": "[v2:127.0.0.1:6812/50430,v1:127.0.0.1:6813/50430]", 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "back_iface": "", 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs": "1", 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:46:02.586 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:46:02.587 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_release": "octopus", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:46:02.588 INFO:tasks.workunit.client.0.smithi118.stdout: "default_device_class": "hdd", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QARN", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": "sda", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "distro": "centos", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:46:02.589 INFO:tasks.workunit.client.0.smithi118.stdout: "distro_version": "8", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "front_addr": "[v2:127.0.0.1:6810/50430,v1:127.0.0.1:6811/50430]", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "front_iface": "", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50430,v1:127.0.0.1:6817/50430]", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50430,v1:127.0.0.1:6815/50430]", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "hostname": "smithi118", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "journal_rotational": "1", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:46:02.590 INFO:tasks.workunit.client.0.smithi118.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_swap_kb": "0", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "mem_total_kb": "32655788", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "os": "Linux", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:46:02.591 INFO:tasks.workunit.client.0.smithi118.stdout: "rotational": "1" 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_first_committed": 1, 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: "osdmap_last_committed": 45, 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: "crushmap": { 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: "devices": [ 2020-03-11T22:46:02.592 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.0", 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:46:02.593 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.1", 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd.2", 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: "class": "hdd" 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.594 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: "types": [ 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 0, 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "osd" 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:46:02.595 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "host" 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 2, 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "chassis" 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 3, 2020-03-11T22:46:02.596 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "rack" 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 4, 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "row" 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 5, 2020-03-11T22:46:02.597 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pdu" 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 6, 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "pod" 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 7, 2020-03-11T22:46:02.598 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "room" 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 8, 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "datacenter" 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.599 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 9, 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "zone" 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 10, 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "region" 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.600 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "root" 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: "buckets": [ 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -1, 2020-03-11T22:46:02.601 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default", 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.602 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.603 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -2, 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "default~hdd", 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 11, 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "root", 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:46:02.604 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.605 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -3, 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118", 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:46:02.606 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.607 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.608 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: "id": -4, 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: "name": "smithi118~hdd", 2020-03-11T22:46:02.609 INFO:tasks.workunit.client.0.smithi118.stdout: "type_id": 1, 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "type_name": "host", 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 19206, 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "alg": "straw2", 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "hash": "rjenkins1", 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.610 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 0, 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 0 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 1, 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.611 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 1 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: "id": 2, 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: "weight": 6402, 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: "pos": 2 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.612 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "rules": [ 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 0, 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "replicated_rule", 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 0, 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 1, 2020-03-11T22:46:02.613 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 1, 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 10, 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:46:02.614 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_firstn", 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:46:02.615 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_id": 1, 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: "rule_name": "test", 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: "ruleset": 1, 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: "type": 3, 2020-03-11T22:46:02.616 INFO:tasks.workunit.client.0.smithi118.stdout: "min_size": 3, 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: "max_size": 3, 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: "steps": [ 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 5 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.617 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "set_choose_tries", 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 100 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "take", 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: "item": -1, 2020-03-11T22:46:02.618 INFO:tasks.workunit.client.0.smithi118.stdout: "item_name": "default" 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "choose_indep", 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 0, 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: "type": "osd" 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.619 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "emit" 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: "tunables": { 2020-03-11T22:46:02.620 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_tries": 0, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_total_tries": 50, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "chooseleaf_stable": 1, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "straw_calc_version": 1, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:46:02.621 INFO:tasks.workunit.client.0.smithi118.stdout: "profile": "jewel", 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "optimal_tunables": 1, 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "legacy_tunables": 0, 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "minimum_required_version": "jewel", 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables": 1, 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables2": 1, 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v2_rules": 1, 2020-03-11T22:46:02.622 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables3": 1, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v3_rules": 0, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v4_buckets": 1, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "require_feature_tunables5": 1, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "has_v5_rules": 0 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "choose_args": {} 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.623 INFO:tasks.workunit.client.0.smithi118.stdout: "fsmap": { 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "default_fscid": -1, 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": { 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "compat": {}, 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "ro_compat": {}, 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "incompat": { 2020-03-11T22:46:02.624 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_1": "base v0.20", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_8": "no anchor table", 2020-03-11T22:46:02.625 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_9": "file layout v2", 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: "feature_flags": { 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: "enable_multiple": false, 2020-03-11T22:46:02.626 INFO:tasks.workunit.client.0.smithi118.stdout: "ever_enabled_multiple": false 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: "standbys": [], 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: "filesystems": [] 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:46:02.627 INFO:tasks.workunit.client.0.smithi118.stdout: "auth": { 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 5, 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "num_secrets": 3 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_active": 1, 2020-03-11T22:46:02.628 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_unknown": 0, 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osd": 3, 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_sum": { 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48430, 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 101, 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 1, 2020-03-11T22:46:02.629 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 303, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 101, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 101, 2020-03-11T22:46:02.630 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:46:02.631 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:46:02.632 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:46:02.633 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:46:02.634 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 0, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 0, 2020-03-11T22:46:02.635 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 18, 2020-03-11T22:46:02.636 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 18, 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 3, 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 3, 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 0 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum": { 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:46:02.637 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3180992, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 35008, 2020-03-11T22:46:02.638 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 9, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145718, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311391808, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318865211392, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:46:02.639 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 35848192, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1555805, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 9486, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221215986 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.640 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:46:02.641 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.642 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_sum_by_class": { 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "hdd": { 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "up_from": 0, 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 0, 2020-03-11T22:46:02.643 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pgs": 3, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "num_osds": 3, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_osds": 3, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "kb": 314572800, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used": 3180992, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_data": 35008, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_omap": 9, 2020-03-11T22:46:02.644 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_used_meta": 3145718, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "kb_avail": 311391808, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "statfs": { 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 322122547200, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 318865211392, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 3221225472, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 35848192, 2020-03-11T22:46:02.645 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1555805, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 9486, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 3221215986 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.646 INFO:tasks.workunit.client.0.smithi118.stdout: "hb_peers": [], 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "num_snap_trimming": 0, 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shards_repaired": 0, 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "op_queue_age_hist": { 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "histogram": [], 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "upper_bound": 1 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.647 INFO:tasks.workunit.client.0.smithi118.stdout: "perf_stat": { 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ms": 0, 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ms": 0, 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "commit_latency_ns": 0, 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "apply_latency_ns": 0 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "alerts": [], 2020-03-11T22:46:02.648 INFO:tasks.workunit.client.0.smithi118.stdout: "network_ping_times": [] 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: "pool_stats": [ 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: "poolid": 1, 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg": 1, 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: "stat_sum": { 2020-03-11T22:46:02.649 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes": 48430, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects": 101, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_clones": 1, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_object_copies": 303, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_missing": 0, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_degraded": 101, 2020-03-11T22:46:02.650 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_misplaced": 0, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_unfound": 0, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_dirty": 101, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_whiteouts": 0, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read": 0, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_read_kb": 0, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write": 101, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_write_kb": 101, 2020-03-11T22:46:02.651 INFO:tasks.workunit.client.0.smithi118.stdout: "num_scrub_errors": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_recovered": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_recovered": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_keys_recovered": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_omap": 0, 2020-03-11T22:46:02.652 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_kb": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_kb": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_promote": 0, 2020-03-11T22:46:02.653 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_high": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_flush_mode_low": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_some": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_evict_mode_full": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_pinned": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_large_omap_objects": 0, 2020-03-11T22:46:02.654 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_manifest": 0, 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_bytes": 0, 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "num_omap_keys": 0, 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "num_objects_repaired": 0 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "store_stats": { 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "total": 0, 2020-03-11T22:46:02.655 INFO:tasks.workunit.client.0.smithi118.stdout: "available": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "internally_reserved": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "allocated": 19857408, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "data_stored": 1241088, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_allocated": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "data_compressed_original": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "omap_allocated": 0, 2020-03-11T22:46:02.656 INFO:tasks.workunit.client.0.smithi118.stdout: "internal_metadata": 0 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: "log_size": 18, 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: "ondisk_log_size": 18, 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: "up": 3, 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: "acting": 3, 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: "num_store_stats": 3 2020-03-11T22:46:02.657 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: "osd_stats": [ 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 176093659146 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.658 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 42949672978 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: "seq": 188978561033 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.659 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_state": [ 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "active+undersized+degraded", 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: "num": 1 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.660 INFO:tasks.workunit.client.0.smithi118.stdout: "num_pg_by_osd": [ 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2, 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.661 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 2147483647, 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 1, 2020-03-11T22:46:02.662 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 0, 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 0, 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: "osd": 0, 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: "num_primary_pg": 1, 2020-03-11T22:46:02.663 INFO:tasks.workunit.client.0.smithi118.stdout: "num_acting_pg": 1, 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": [ 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: "pool": 1, 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: "purged_snaps": {} 2020-03-11T22:46:02.664 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: "servicemap": { 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": 1, 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: "modified": "2020-03-11T22:43:33.487085+0000", 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: "services": {} 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:46:02.665 INFO:tasks.workunit.client.0.smithi118.stdout: "paxos": { 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout: "first_committed": 1, 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout: "last_committed": 428, 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout: "last_pn": 0, 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout: "accepted_pn": 0 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:46:02.666 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:46:02.667 INFO:tasks.workunit.client.0.smithi118.stderr:report 240041997 2020-03-11T22:46:02.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:46:02.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:46:02.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:02.668 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery 2020-03-11T22:46:02.832 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:02.832 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:28.154486+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:02.832 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:02.832 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:02.833 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:02.841 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:02.842 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:46:02.842 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:46:02.842 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.0.asok 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:02.843 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:46:02.934 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:46:02.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:02.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:46:02.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.2.asok 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:46:03.033 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:46:03.040 INFO:tasks.workunit.client.0.smithi118.stdout:revive divergent 1 2020-03-11T22:46:03.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:46:03.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:03.356 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:03.357 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:28.154486+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:03.357 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:03.357 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:03.358 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:03.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:46:04.065 INFO:tasks.workunit.client.0.smithi118.stderr:noup is set 2020-03-11T22:46:04.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:46:04.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:46:04.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:04.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:04.078 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:46:04.079 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:04.080 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:04.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:46:04.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:46:04.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:04.081 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:46:04.081 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:46:04.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:46:04.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:04.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:04.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:04.102 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:04.100+0000 7f531bd9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:04.104 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:04.101+0000 7f531bd9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:04.106 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:04.103+0000 7f531bd9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:04.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:46:04.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:46:04.625 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:04.622+0000 7f531bd9aec0 -1 Falling back to public interface 2020-03-11T22:46:05.448 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:05.445+0000 7f531bd9aec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:46:09.424 INFO:tasks.workunit.client.0.smithi118.stdout:delay recovery divergent 2020-03-11T22:46:09.424 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:46:09.424 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:09.751 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:09.751 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:28.154486+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:09.751 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:09.751 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:09.752 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:09.761 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:09.762 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17426/ceph-osd.1.asok 2020-03-11T22:46:09.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:09.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17426/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:46:09.852 INFO:tasks.workunit.client.0.smithi118.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:46:09.859 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:46:10.687 INFO:tasks.workunit.client.0.smithi118.stderr:noup is unset 2020-03-11T22:46:10.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:10.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:10.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:10.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:10.697 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:11.061 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 41 up_thru 44 down_at 40 last_clean_interval [35,39) [v2:127.0.0.1:6802/50198,v1:127.0.0.1:6803/50198] [v2:127.0.0.1:6804/50198,v1:127.0.0.1:6805/50198] exists,up 89e499ae-fb1d-4d02-a7e0-88539fbd14b9 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:46:11.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:11.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:11.421 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/51826,v1:127.0.0.1:6819/51826] [v2:127.0.0.1:6820/51826,v1:127.0.0.1:6821/51826] exists,up bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:46:11.421 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:11.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:11.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:11.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:11.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:46:11.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:11.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:11.424 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:11.760 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 44 up_thru 0 down_at 43 last_clean_interval [36,39) [v2:127.0.0.1:6810/50430,v1:127.0.0.1:6811/50430] [v2:127.0.0.1:6812/50430,v1:127.0.0.1:6813/50430] exists,up b7258573-0afc-4577-85d0-d4c70cda4ce8 2020-03-11T22:46:11.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:11.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:11.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:11.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:12.079 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:12.079 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:28.154486+0000 39'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:12.079 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:12.079 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:12.081 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:12.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:46:12.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:12.089 INFO:tasks.workunit.client.0.smithi118.stdout:wait for peering 2020-03-11T22:46:12.410 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:12.410 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:28.154486+0000 39'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:12.410 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:12.411 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:12.412 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:12.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:46:24.214 INFO:tasks.workunit.client.0.smithi118.stdout:killing divergent 1 2020-03-11T22:46:24.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:46:24.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:24.547 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:24.547 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:46:11.331689+0000 39'102 51:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:24.547 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:24.547 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:24.548 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:24.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:46:24.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:24.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:24.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:24.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:24.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:24.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:24.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:46:24.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:24.663 INFO:tasks.workunit.client.0.smithi118.stdout:reviving divergent 1 2020-03-11T22:46:24.980 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:24.981 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:46:11.331689+0000 39'102 51:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:24.981 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:24.981 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:24.982 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:24.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:46:24.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:46:24.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:24.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:46:24.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:24.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:24.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:24.996 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:24.997 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:24.997 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:24.997 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:24.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:24.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:24.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:24.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:46:24.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:46:24.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=9298b8d8-9f52-4152-9823-0eb7fba47eea --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17426/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:24.999 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:46:25.000 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:46:25.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:46:25.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:25.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:25.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:25.019 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:25.015+0000 7f79460aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.021 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:25.017+0000 7f79460aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.023 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:25.019+0000 7f79460aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:46:25.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:25.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:25.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:25.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:25.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:25.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:25.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:25.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:25.344 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:25.538 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:25.535+0000 7f79460aeec0 -1 Falling back to public interface 2020-03-11T22:46:25.739 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/51826,v1:127.0.0.1:6819/51826] [v2:127.0.0.1:6820/51826,v1:127.0.0.1:6821/51826] exists,up bdbe9782-9f6f-479f-8860-f1efe9d4e8aa 2020-03-11T22:46:25.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:25.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:25.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:25.740 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:46:26.325 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:26.321+0000 7f79460aeec0 -1 osd.1 51 log_to_monitors {default=true} 2020-03-11T22:46:45.740 INFO:tasks.workunit.client.0.smithi118.stdout:allowing recovery 2020-03-11T22:46:45.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:46:45.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:46.064 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:46.064 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:46:27.531527+0000 51'103 54:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:46.064 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:46.065 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:46.066 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:46.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:46.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:46:46.195 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:46.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:46.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:46:46.323 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:46.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:46.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:46:46.451 INFO:tasks.workunit.client.0.smithi118.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:46.461 INFO:tasks.workunit.client.0.smithi118.stdout:reading divergent objects 2020-03-11T22:46:46.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:46:46.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:46.779 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:46.779 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:46:27.531527+0000 51'103 54:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:55.877102+0000 0'0 2020-03-11T22:43:55.877102+0000 0 2020-03-11T22:46:46.779 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:46:46.780 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:46.781 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:46:46.789 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:46:46.790 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:46:46.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:46.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:46:46.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:46.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:46:46.862 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:46.862 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:46:46.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:46.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:46:46.918 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:46:46.919 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:46.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:46.924 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:44:17.331823+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.924 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:44:17.331823+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.924 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:46.924 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:44:23.848003+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:44:23.848003+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:46:46.925 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:46:46.926 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:46.926 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2020-03-11T22:46:46.926 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:44:23.848003+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.926 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:46:46.926 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:44:22.727918+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:44:23.730864+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.927 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:44:22.727918+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.927 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:44:23.730864+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.927 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:44:23.730864+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:46.928 INFO:tasks.workunit.client.0.smithi118.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:11.307+0000 7f52f9f55700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:46:46.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:46.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:46.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:46:46.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:46:46.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:46:46.931 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:46:46.931 INFO:tasks.workunit.client.0.smithi118.stdout:success 2020-03-11T22:46:47.409 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' does not exist 2020-03-11T22:46:47.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:47.419 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:47.419 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:47.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:47.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:46:47.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:47.526 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:47.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:47.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:47.531 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:47.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:47.532 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:47.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:47.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:47.533 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:47.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:47.535 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:47.535 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:47.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:47.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:47.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:47.550 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:47.550 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:47.550 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:46:47.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:47.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:47.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:47.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:47.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:47.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:47.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:47.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:47.554 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:47.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:47.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:47.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:47.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:47.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:47.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:46:47.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:47.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:47.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:47.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17426 2020-03-11T22:46:47.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17426 2020-03-11T22:46:47.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:47.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:47.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:46:47.562 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:46:47.562 INFO:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:46:47.669 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:46:47.670 INFO:teuthology.orchestra.run.smithi118:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:46:47.700 INFO:tasks.workunit.client.0.smithi118.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:46:47.707 INFO:tasks.workunit.client.0.smithi118.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:46:47.708 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:46:47.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none ' 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:46:47.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:46:47.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:46:47.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:46:47.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:46:47.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:46:47.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:46:47.713 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:47.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:47.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:47.718 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:47.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52754 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:47.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:47.721 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52754 2020-03-11T22:46:47.722 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:47.722 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:46:47.722 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:46:47.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:47.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:47.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:47.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:46:47.724 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:46:48.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:48.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:48.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:48.224 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:48.225 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52754/ceph-mon.a.asok 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:48.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52754/ceph-mon.a.asok config get fsid 2020-03-11T22:46:48.312 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:48.312 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:48.312 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:48.312 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:48.313 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:48.314 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:48.314 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52754/ceph-mon.a.asok 2020-03-11T22:46:48.314 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:48.314 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52754/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:46:48.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:49.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:49.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:49.306 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:49.306 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:49.306 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:49.307 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:49.307 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:49.307 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:49.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:49.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:49.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:49.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:49.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:49.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:46:49.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:49.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bbc2bf64-d274-4c84-80cf-9bab6b10a385 2020-03-11T22:46:49.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 bbc2bf64-d274-4c84-80cf-9bab6b10a385' 2020-03-11T22:46:49.334 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:49.334 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 bbc2bf64-d274-4c84-80cf-9bab6b10a385 2020-03-11T22:46:49.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBZamlejzswFBAA5KDlWvqll5pQafm2YoRrzQ== 2020-03-11T22:46:49.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBZamlejzswFBAA5KDlWvqll5pQafm2YoRrzQ=="}' 2020-03-11T22:46:49.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bbc2bf64-d274-4c84-80cf-9bab6b10a385 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:46:49.681 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:49.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:46:49.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZamlejzswFBAA5KDlWvqll5pQafm2YoRrzQ== --osd-uuid bbc2bf64-d274-4c84-80cf-9bab6b10a385 2020-03-11T22:46:49.711 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.708+0000 7f5695305ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:49.713 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.710+0000 7f5695305ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:49.714 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.712+0000 7f5695305ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:49.715 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.712+0000 7f5695305ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:49.715 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.712+0000 7f5695305ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:49.715 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.712+0000 7f5695305ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:49.749 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:49.746+0000 7f5695305ec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:46:52.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:46:52.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:52.765 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:46:52.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:52.766 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:53.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:53.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:53.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:53.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:53.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:53.211 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:46:53.228 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:53.224+0000 7f1a90114ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:53.252 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:53.248+0000 7f1a90114ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:53.254 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:53.250+0000 7f1a90114ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:53.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:53.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:53.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:53.564 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:53.769 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:53.765+0000 7f1a90114ec0 -1 Falling back to public interface 2020-03-11T22:46:53.905 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:54.487+0000 7f1a90114ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:55.013 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:46:55.260 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53183,v1:127.0.0.1:6803/53183] [v2:127.0.0.1:6804/53183,v1:127.0.0.1:6805/53183] exists,up bbc2bf64-d274-4c84-80cf-9bab6b10a385 2020-03-11T22:46:55.260 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 2020-03-11T22:46:55.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:55.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:55.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:55.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:46:55.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:55.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:55.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:55.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:55.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:46:55.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:55.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:55.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:55.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:55.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:46:55.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:46:55.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:55.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:55.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:55.265 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:55.265 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:55.265 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:55.265 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:55.266 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:55.266 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:46:55.266 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:55.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:46:55.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:55.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:55.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:55.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:55.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:55.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:55.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:55.268 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:55.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:55.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:55.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:46:55.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:55.269 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 2020-03-11T22:46:55.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 c15eb849-52c8-4fa2-aca0-d8e3e7d7f197' 2020-03-11T22:46:55.270 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:55.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBfamle0u7FDxAA+kn41H9nyzJTkLN2RLX/9Q== 2020-03-11T22:46:55.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBfamle0u7FDxAA+kn41H9nyzJTkLN2RLX/9Q=="}' 2020-03-11T22:46:55.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:46:55.765 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:46:55.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:46:55.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBfamle0u7FDxAA+kn41H9nyzJTkLN2RLX/9Q== --osd-uuid c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 2020-03-11T22:46:55.795 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.791+0000 7f8c80daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:55.806 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.803+0000 7f8c80daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:55.808 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.804+0000 7f8c80daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:55.809 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.805+0000 7f8c80daeec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:55.809 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.805+0000 7f8c80daeec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:55.809 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.805+0000 7f8c80daeec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:55.917 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:55.911+0000 7f8c80daeec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:46:59.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:46:59.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:59.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:59.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:59.004 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:46:59.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:59.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:59.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:59.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:59.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:59.454 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:46:59.471 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:59.467+0000 7fd73911aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:59.473 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:59.469+0000 7fd73911aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:59.475 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:59.471+0000 7fd73911aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:59.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:59.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:59.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:59.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:59.802 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:46:59.990 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:46:59.986+0000 7fd73911aec0 -1 Falling back to public interface 2020-03-11T22:47:00.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:00.763 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:00.759+0000 7fd73911aec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:47:01.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:01.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:01.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:01.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:01.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:01.139 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:01.505 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53549,v1:127.0.0.1:6811/53549] [v2:127.0.0.1:6812/53549,v1:127.0.0.1:6813/53549] exists,up c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 2020-03-11T22:47:01.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:01.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:01.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:01.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:01.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:01.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:01.508 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:01.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:01.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:01.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:01.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:01.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:01.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:47:01.511 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:01.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1990933b-cbd3-4f47-bae4-2f5ea6dad138 2020-03-11T22:47:01.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 1990933b-cbd3-4f47-bae4-2f5ea6dad138' 2020-03-11T22:47:01.512 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:01.512 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 1990933b-cbd3-4f47-bae4-2f5ea6dad138 2020-03-11T22:47:01.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBlamlem6W/HhAAjNXC4AGs9+6rvzd8vWj8/A== 2020-03-11T22:47:01.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBlamlem6W/HhAAjNXC4AGs9+6rvzd8vWj8/A=="}' 2020-03-11T22:47:01.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1990933b-cbd3-4f47-bae4-2f5ea6dad138 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:47:01.992 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:47:02.002 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:47:02.002 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBlamlem6W/HhAAjNXC4AGs9+6rvzd8vWj8/A== --osd-uuid 1990933b-cbd3-4f47-bae4-2f5ea6dad138 2020-03-11T22:47:02.022 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.018+0000 7f09345cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:02.024 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.020+0000 7f09345cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:02.025 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.022+0000 7f09345cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:02.026 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.022+0000 7f09345cdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:47:02.026 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.022+0000 7f09345cdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:47:02.026 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.022+0000 7f09345cdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:47:02.095 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:02.092+0000 7f09345cdec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:47:05.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:47:05.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:05.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:47:05.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:05.182 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:47:05.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:47:05.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:05.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:05.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:05.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:05.634 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:47:05.650 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:05.647+0000 7fd9e365eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:05.652 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:05.649+0000 7fd9e365eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:05.654 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:05.650+0000 7fd9e365eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:05.973 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:05.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:05.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:05.974 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:05.974 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:06.175 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:06.171+0000 7fd9e365eec0 -1 Falling back to public interface 2020-03-11T22:47:06.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:06.944 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:06.941+0000 7fd9e365eec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:47:07.314 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:07.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:07.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:07.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:07.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:07.315 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:07.696 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/53904,v1:127.0.0.1:6819/53904] [v2:127.0.0.1:6820/53904,v1:127.0.0.1:6821/53904] exists,up 1990933b-cbd3-4f47-bae4-2f5ea6dad138 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:07.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:07.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:07.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:07.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:07.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:07.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:47:07.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:07.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=880806a2-eb4c-4b58-a51c-bca632369984 2020-03-11T22:47:07.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 880806a2-eb4c-4b58-a51c-bca632369984' 2020-03-11T22:47:07.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:07.703 INFO:tasks.workunit.client.0.smithi118.stdout:add osd3 880806a2-eb4c-4b58-a51c-bca632369984 2020-03-11T22:47:07.710 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBramle1NsyKhAAhJxbmxgMxnGLXeG+7oCKWg== 2020-03-11T22:47:07.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBramle1NsyKhAAhJxbmxgMxnGLXeG+7oCKWg=="}' 2020-03-11T22:47:07.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 880806a2-eb4c-4b58-a51c-bca632369984 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:47:08.148 INFO:tasks.workunit.client.0.smithi118.stdout:3 2020-03-11T22:47:08.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:47:08.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBramle1NsyKhAAhJxbmxgMxnGLXeG+7oCKWg== --osd-uuid 880806a2-eb4c-4b58-a51c-bca632369984 2020-03-11T22:47:08.178 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.174+0000 7f1fa14e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.179 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.176+0000 7f1fa14e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.180 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.177+0000 7f1fa14e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.181 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.177+0000 7f1fa14e6ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:08.181 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.177+0000 7f1fa14e6ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:08.181 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.177+0000 7f1fa14e6ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:08.226 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:08.224+0000 7f1fa14e6ec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:47:11.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:47:11.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:11.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:47:11.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:11.529 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd3 key to auth repository 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:11.980 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.3 2020-03-11T22:47:11.998 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:11.995+0000 7f3778579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:11.999 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:11.997+0000 7f3778579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:12.001 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:11.998+0000 7f3778579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:12.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:12.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:12.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:12.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:12.333 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:12.516 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:12.513+0000 7f3778579ec0 -1 Falling back to public interface 2020-03-11T22:47:12.673 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:13.301 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:13.298+0000 7f3778579ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:13.675 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:14.024 INFO:tasks.workunit.client.0.smithi118.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54259,v1:127.0.0.1:6827/54259] [v2:127.0.0.1:6828/54259,v1:127.0.0.1:6829/54259] exists,up 880806a2-eb4c-4b58-a51c-bca632369984 2020-03-11T22:47:14.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:14.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:14.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:14.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:47:14.587 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:47:15.244 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'ec' already exists 2020-03-11T22:47:15.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:47:16.501 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 53904 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54514 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:47:17.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54537' 2020-03-11T22:47:17.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:47:17.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54538' 2020-03-11T22:47:17.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:47:17.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54539' 2020-03-11T22:47:17.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:47:17.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:47:18.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:47:18.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:18.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:18.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:18.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:18.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:18.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:18.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54514 54537 54538 54539 2020-03-11T22:47:18.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:47:18.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:47:18.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:18.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:18.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:18.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:18.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:18.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:18.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:47:18.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:47:18.624 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:47:18.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:18.625 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:47:18.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:47:18.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:18.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:18.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:18.640 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:18.637+0000 7f490a3a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:18.646 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:18.643+0000 7f490a3a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:18.648 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:18.645+0000 7f490a3a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:18.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:47:18.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:18.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:18.985 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:18.986 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:19.160+0000 7f490a3a3ec0 -1 Falling back to public interface 2020-03-11T22:47:19.327 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53183,v1:127.0.0.1:6803/53183] [v2:127.0.0.1:6804/53183,v1:127.0.0.1:6805/53183] exists,up bbc2bf64-d274-4c84-80cf-9bab6b10a385 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:47:19.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:19.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:19.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:19.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:19.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:19.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:19.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:19.344 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:19.341+0000 7f3a8980fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.346 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:19.343+0000 7f3a8980fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.347 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:19.345+0000 7f3a8980fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:19.865 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:19.862+0000 7f3a8980fec0 -1 Falling back to public interface 2020-03-11T22:47:20.046 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53549,v1:127.0.0.1:6811/53549] [v2:127.0.0.1:6812/53549,v1:127.0.0.1:6813/53549] exists,up c15eb849-52c8-4fa2-aca0-d8e3e7d7f197 2020-03-11T22:47:20.046 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:47:20.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:47:20.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:20.049 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:20.050 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:47:20.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:47:20.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:47:20.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:20.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:20.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:20.062 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.060+0000 7f14010b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.064 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.061+0000 7f14010b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.065 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.062+0000 7f14010b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.091 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.088+0000 7f490a3a3ec0 -1 osd.0 23 log_to_monitors {default=true} 2020-03-11T22:47:20.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:47:20.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:20.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:20.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:20.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:20.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:20.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:20.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:20.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:20.422 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:20.580 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.578+0000 7f14010b8ec0 -1 Falling back to public interface 2020-03-11T22:47:20.876 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/53904,v1:127.0.0.1:6819/53904] [v2:127.0.0.1:6820/53904,v1:127.0.0.1:6821/53904] exists,up 1990933b-cbd3-4f47-bae4-2f5ea6dad138 2020-03-11T22:47:20.876 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.3 2020-03-11T22:47:20.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:20.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:47:20.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:20.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:47:20.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:20.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:20.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:20.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:20.879 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:20.880 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:20.880 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:20.880 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:20.880 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:20.881 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:20.881 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:20.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' 2020-03-11T22:47:20.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:20.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:20.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=ebb5b895-8ba5-487e-af45-c1b4a1944219 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:20.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:20.901 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.898+0000 7ffa5b330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.902 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.900+0000 7ffa5b330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.903 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.900+0000 7ffa5b330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.986 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:20.983+0000 7f3a8980fec0 -1 osd.1 23 log_to_monitors {default=true} 2020-03-11T22:47:21.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:21.257 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:21.257 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:21.419 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:21.416+0000 7ffa5b330ec0 -1 Falling back to public interface 2020-03-11T22:47:21.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:21.889 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:21.886+0000 7f14010b8ec0 -1 osd.2 22 log_to_monitors {default=true} 2020-03-11T22:47:22.692 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:22.689+0000 7ffa5b330ec0 -1 osd.3 23 log_to_monitors {default=true} 2020-03-11T22:47:22.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:22.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:22.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:22.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:22.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:22.700 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:23.113 INFO:tasks.workunit.client.0.smithi118.stdout:osd.3 up in weight 1 up_from 33 up_thru 0 down_at 26 last_clean_interval [18,23) [v2:127.0.0.1:6826/55059,v1:127.0.0.1:6827/55059] [v2:127.0.0.1:6828/55059,v1:127.0.0.1:6829/55059] exists,up 880806a2-eb4c-4b58-a51c-bca632369984 2020-03-11T22:47:23.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:23.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:23.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:23.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:47:23.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:23.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:23.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:23.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:23.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:23.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:23.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:23.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:23.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:23.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:23.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:23.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:23.203 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:23.547 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:23.547 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:47:23.547 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:47:23.547 INFO:tasks.workunit.client.0.smithi118.stderr:3' 2020-03-11T22:47:23.547 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:23.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:23.548 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:23.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182403 2020-03-11T22:47:23.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-107374182403' 2020-03-11T22:47:23.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:23.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:23.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084291 2020-03-11T22:47:23.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-107374182403 1-120259084291' 2020-03-11T22:47:23.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:23.799 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:47:23.918 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:47:23.918 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-107374182403 1-120259084291 2-133143986179' 2020-03-11T22:47:23.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:23.919 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:47:24.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:47:24.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-107374182403 1-120259084291 2-133143986179 3-141733920771' 2020-03-11T22:47:24.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:24.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-107374182403 2020-03-11T22:47:24.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:24.041 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:24.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-107374182403 2020-03-11T22:47:24.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:24.043 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182403 2020-03-11T22:47:24.043 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 107374182403' 2020-03-11T22:47:24.043 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 107374182403 2020-03-11T22:47:24.044 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:24.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 107374182403 2020-03-11T22:47:24.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:25.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:25.380 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:25.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182403 -lt 107374182403 2020-03-11T22:47:25.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:25.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-120259084291 2020-03-11T22:47:25.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:25.733 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:25.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-120259084291 2020-03-11T22:47:25.734 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:25.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084291 2020-03-11T22:47:25.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 120259084291' 2020-03-11T22:47:25.735 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 120259084291 2020-03-11T22:47:25.736 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:26.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084291 -lt 120259084291 2020-03-11T22:47:26.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:26.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-133143986179 2020-03-11T22:47:26.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:26.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:47:26.106 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-133143986179 2020-03-11T22:47:26.106 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:26.106 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:47:26.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2020-03-11T22:47:26.107 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 133143986179 2020-03-11T22:47:26.107 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:26.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986179 -lt 133143986179 2020-03-11T22:47:26.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:26.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-141733920771 2020-03-11T22:47:26.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:26.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:47:26.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-141733920771 2020-03-11T22:47:26.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:26.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:47:26.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2020-03-11T22:47:26.466 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 141733920771 2020-03-11T22:47:26.466 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:47:26.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:47:26.808 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:26.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:26.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:27.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:47:27.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:27.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:27.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:27.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:27.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:27.247 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:27.248 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:27.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:27.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:27.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:27.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:47:28.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:47:28.020 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:47:28.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:28.021 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:28.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:28.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:28.021 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:28.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:28.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:28.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:28.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:28.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:28.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:28.138 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:28.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:28.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:28.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.141 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:28.141 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:47:28.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:47:28.147 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:28.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:28.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52754 2020-03-11T22:47:28.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:28.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:28.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:28.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:28.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:28.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:28.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:28.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:28.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:28.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:28.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:28.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:28.153 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:28.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:28.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:28.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:47:28.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:28.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52754 2020-03-11T22:47:28.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52754 2020-03-11T22:47:28.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:28.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:28.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:47:28.159 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:47:28.159 INFO:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:47:28.234 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:47:28.234 INFO:teuthology.orchestra.run.smithi118:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:47:28.285 INFO:tasks.workunit.client.0.smithi118.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:47:28.295 INFO:tasks.workunit.client.0.smithi118.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:47:28.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:47:28.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:47:28.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:47:28.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:47:28.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none ' 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:47:28.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:47:28.298 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:47:28.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:47:28.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:47:28.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:47:28.299 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:47:28.299 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_backfill_priority' 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:47:28.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:28.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:28.304 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:28.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:28.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:28.305 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:28.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:47:28.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:47:28.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55883 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:47:28.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.55883 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:47:28.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:47:28.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:47:28.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:47:28.629 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:47:28.629 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:28.629 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:28.629 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:28.630 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.630 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:28.630 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:28.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:47:28.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:47:28.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:47:28.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:47:28.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:47:28.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.832 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:28.833 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-mon.a.asok 2020-03-11T22:47:28.833 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:47:28.833 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55883/ceph-mon.a.asok config get fsid 2020-03-11T22:47:28.917 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:47:28.917 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:47:28.918 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:28.919 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.919 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:28.919 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-mon.a.asok 2020-03-11T22:47:28.919 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:47:28.919 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55883/ceph-mon.a.asok config get mon_host 2020-03-11T22:47:29.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:47:29.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:47:29.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:29.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:29.912 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:29.912 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:29.912 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:29.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:29.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:47:29.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:47:29.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:47:29.935 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:47:29.936 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:47:29.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:29.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:47:29.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:29.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:29.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:29.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:29.944 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f17a3866-9ca7-4c6f-ac9f-76405130b83a 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 f17a3866-9ca7-4c6f-ac9f-76405130b83a' 2020-03-11T22:47:29.945 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:29.946 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 f17a3866-9ca7-4c6f-ac9f-76405130b83a 2020-03-11T22:47:29.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCBamlePLfSOBAAXNtwUet0RhwbABPkOKlsmw== 2020-03-11T22:47:29.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCBamlePLfSOBAAXNtwUet0RhwbABPkOKlsmw=="}' 2020-03-11T22:47:29.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f17a3866-9ca7-4c6f-ac9f-76405130b83a -i td/osd-backfill-prio/0/new.json 2020-03-11T22:47:30.295 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:30.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:47:30.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCBamlePLfSOBAAXNtwUet0RhwbABPkOKlsmw== --osd-uuid f17a3866-9ca7-4c6f-ac9f-76405130b83a 2020-03-11T22:47:30.331 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.328+0000 7f0a763d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.330+0000 7f0a763d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.331+0000 7f0a763d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.334 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.332+0000 7f0a763d5ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:30.334 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.332+0000 7f0a763d5ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:30.335 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.332+0000 7f0a763d5ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:30.379 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:30.376+0000 7f0a763d5ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:47:33.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:47:33.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:33.396 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:47:33.396 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:33.396 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:47:33.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:47:33.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:33.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:33.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:33.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:33.874 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:47:33.897 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:33.894+0000 7f03f8173ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:33.898 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:33.896+0000 7f03f8173ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:33.900 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:33.897+0000 7f03f8173ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:34.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:47:34.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:34.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:34.222 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:34.416 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:34.413+0000 7f03f8173ec0 -1 Falling back to public interface 2020-03-11T22:47:34.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:35.206 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:35.203+0000 7f03f8173ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:47:35.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:35.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:35.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:35.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:35.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:35.571 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:35.952 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56315,v1:127.0.0.1:6803/56315] [v2:127.0.0.1:6804/56315,v1:127.0.0.1:6805/56315] exists,up f17a3866-9ca7-4c6f-ac9f-76405130b83a 2020-03-11T22:47:35.952 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 270f9be8-f1a0-440a-9cc7-c15f862f8e02 2020-03-11T22:47:35.952 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:35.952 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:35.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:35.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:35.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=270f9be8-f1a0-440a-9cc7-c15f862f8e02 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 270f9be8-f1a0-440a-9cc7-c15f862f8e02' 2020-03-11T22:47:35.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:35.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCHamleQbSBORAAONBLS7Fo8mAn//EJ9on3iQ== 2020-03-11T22:47:35.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCHamleQbSBORAAONBLS7Fo8mAn//EJ9on3iQ=="}' 2020-03-11T22:47:35.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 270f9be8-f1a0-440a-9cc7-c15f862f8e02 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:47:36.501 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:36.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:47:36.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCHamleQbSBORAAONBLS7Fo8mAn//EJ9on3iQ== --osd-uuid 270f9be8-f1a0-440a-9cc7-c15f862f8e02 2020-03-11T22:47:36.537 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.533+0000 7f232f7a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.538 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.535+0000 7f232f7a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.540 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.536+0000 7f232f7a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.541 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.537+0000 7f232f7a9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:36.541 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.537+0000 7f232f7a9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:36.541 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.537+0000 7f232f7a9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:36.626 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:36.623+0000 7f232f7a9ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:47:39.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:47:39.735 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:39.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:47:39.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:39.736 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:47:40.194 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:47:40.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:47:40.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:40.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:40.195 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:40.195 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:40.218 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:40.214+0000 7f62d309bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:40.220 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:40.216+0000 7f62d309bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:40.221 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:40.217+0000 7f62d309bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:40.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:47:40.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:40.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:40.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:40.542 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:47:40.736 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:40.732+0000 7f62d309bec0 -1 Falling back to public interface 2020-03-11T22:47:40.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:41.537 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:47:41.534+0000 7f62d309bec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:47:41.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:41.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:41.968 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:41.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:41.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:41.969 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:47:42.316 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56670,v1:127.0.0.1:6811/56670] [v2:127.0.0.1:6812/56670,v1:127.0.0.1:6813/56670] exists,up 270f9be8-f1a0-440a-9cc7-c15f862f8e02 2020-03-11T22:47:42.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:42.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:42.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:42.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:42.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:42.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:47:42.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:47:42.813 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test1' already exists 2020-03-11T22:47:42.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:43.822 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:44.600 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 2 2020-03-11T22:47:44.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:44.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:47:44.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:47:45.194 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test2' already exists 2020-03-11T22:47:45.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:46.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:46.921 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 2 size to 2 2020-03-11T22:47:46.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:46.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:47:46.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:47:47.420 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test3' already exists 2020-03-11T22:47:47.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:48.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:47:49.192 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 3 size to 2 2020-03-11T22:47:49.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:47:54.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:54.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:54.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:54.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:54.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:54.204 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:54.204 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:54.205 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:54.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:54.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:54.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:54.305 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:1' 2020-03-11T22:47:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:54.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:47:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:47:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:54.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:47:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2020-03-11T22:47:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:54.902 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:47:54.902 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:54.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:54.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:47:54.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:54.905 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:47:54.905 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:47:54.905 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:47:54.906 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:55.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:47:55.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:56.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:56.353 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:56.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:47:56.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:56.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:47:56.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:56.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:56.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:47:56.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:56.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:47:56.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:47:56.610 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:47:56.611 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:56.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:47:56.954 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:56.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:56.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:57.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:47:57.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:57.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:57.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:47:57.763 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:57.764 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:57.764 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:58.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:47:58.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:47:58.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:47:58.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:47:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:49.356325+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:47.232170+0000 0'0 2020-03-11T22:47:47.232170+0000 0 2020-03-11T22:47:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:44.514314+0000 0'0 27:28 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:42.599888+0000 0'0 2020-03-11T22:47:42.599888+0000 0 2020-03-11T22:47:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:47.441398+0000 0'0 26:22 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:47:44.937864+0000 0'0 2020-03-11T22:47:44.937864+0000 0 2020-03-11T22:47:58.559 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:47:58.560 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:58.560 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:47:58.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:47:58.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:47:58.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:47:58.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:47:58.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:47:58.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:58.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:58.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:47:58.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:47:58.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:47:58.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:47:58.905 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:47:58.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:47:58.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:47:58.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:58.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:47:58.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:47:59.269 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:47:59.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:47:59.271 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:47:59.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:47:59.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:47:59.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:47:59.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:47:59.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:47:59.276 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:59.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:59.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:47:59.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:59.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:47:59.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:59.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:47:59.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:47:59.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:47:59.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:47:59.786 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test3' does not exist 2020-03-11T22:47:59.793 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:47:59.794 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:47:59.794 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:47:59.795 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:47:59.796 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:47:59.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:47:59.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:48:00.422 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 1 2020-03-11T22:48:00.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:48:01.333 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:48:01.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:48:02.101 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 2 size to 1 2020-03-11T22:48:02.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:48:02.719 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:48:02.727 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:48:02.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:48:02.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:48:02.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:48:02.728 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:48:02.729 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:48:02.729 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:48:02.729 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:48:02.730 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:48:02.730 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:48:02.730 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:48:02.818 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:48:02.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:48:02.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:48:02.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:48:02.819 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:48:03.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:48:03.162 INFO:tasks.workunit.client.0.smithi118.stderr:1' 2020-03-11T22:48:03.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:48:03.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:03.163 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:48:03.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-03-11T22:48:03.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491' 2020-03-11T22:48:03.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:03.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:48:03.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-03-11T22:48:03.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2020-03-11T22:48:03.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:03.417 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836491 2020-03-11T22:48:03.417 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:03.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:03.419 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836491 2020-03-11T22:48:03.419 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:03.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836491 2020-03-11T22:48:03.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-03-11T22:48:03.420 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836491 2020-03-11T22:48:03.421 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:03.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836491 2020-03-11T22:48:03.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:04.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:04.764 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:05.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836491 2020-03-11T22:48:05.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:06.116 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:48:06.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836491 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672970 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:06.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:06.468 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672970 2020-03-11T22:48:06.468 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:06.469 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672970 2020-03-11T22:48:06.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-03-11T22:48:06.470 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672970 2020-03-11T22:48:06.470 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:06.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672970 2020-03-11T22:48:06.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:06.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:06.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:07.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:48:07.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:07.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:07.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:07.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:07.301 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:07.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:07.302 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:07.634 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:48:07.635 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:07.635 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:07.636 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:08.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:48:08.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:08.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:08.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:48:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:10+0 records in 2020-03-11T22:48:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:10+0 records out 2020-03-11T22:48:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0460587 s, 228 MB/s 2020-03-11T22:48:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:48:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:48:08.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:48:08.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.792 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.792 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.736 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.636 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.137 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.361 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.362 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.141 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.979 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.821 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.822 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:48:15.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:48:15.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.413 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:48:15.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.663 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:48:15.913 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.914 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:48:16.206 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.207 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:48:16.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:48:16.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:48:16.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:48:17.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:48:17.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:48:17.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:48:17.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:48:18.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:48:18.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:48:18.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.804 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:48:19.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:48:19.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.305 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:48:19.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:48:19.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.858 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:48:21.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:48:21.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:48:21.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:48:21.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:48:21.511 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:48:21.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:48:21.781 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.781 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.099 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.099 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.249 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.250 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.767 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.050 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.323 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.901 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.951 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.952 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:48:29.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:29.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:48:29.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:29.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:48:29.795 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:29.795 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:48:30.087 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.088 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:48:30.335 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.335 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:48:30.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:48:30.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:48:31.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:31.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:48:31.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:31.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:48:32.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:48:32.419 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.420 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:48:32.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:48:32.936 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.936 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:48:33.178 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:33.178 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:48:33.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:33.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:48:33.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:33.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:48:34.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:48:34.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:48:34.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:48:34.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:48:35.072 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:48:35.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:48:35.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:48:35.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:48:35.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:48:35.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:48:35.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:48:35.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:48:35.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:48:35.075 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:48:35.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:48:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:48:35.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:48:35.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:48:35.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:48:36.101 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:36.101 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:00.564977+0000 39'150 39:194 [1] 1 [1] 1 0'0 2020-03-11T22:47:42.599888+0000 0'0 2020-03-11T22:47:42.599888+0000 0 2020-03-11T22:48:36.102 INFO:tasks.workunit.client.0.smithi118.stdout:2.0 38 0 0 0 0 392167424 0 0 12 12 active+clean 2020-03-11T22:48:01.913437+0000 39'112 39:151 [0] 0 [0] 0 0'0 2020-03-11T22:47:44.937864+0000 0'0 2020-03-11T22:47:44.937864+0000 0 2020-03-11T22:48:36.102 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:48:36.102 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:36.103 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:48:36.111 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:48:36.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:48:36.808 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 2 2020-03-11T22:48:36.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:48:37.427 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 2 size to 2 2020-03-11T22:48:37.435 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:48:42.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:48:42.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:48:42.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:48:42.438 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:42.438 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:42.439 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:48:42.439 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:48:42.439 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:48:42.439 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:48:42.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:48:42.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:42.537 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:48:42.538 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 152, 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:48:42.539 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "2.0", 2020-03-11T22:48:42.540 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 153, 2020-03-11T22:48:42.541 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:48:42.541 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.541 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:48:42.541 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.541 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:48:42.542 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.0.asok 2020-03-11T22:48:42.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:48:42.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:48:42.634 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:48:42.634 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:42.634 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:48:42.635 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "2.0", 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 153, 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:48:42.636 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 152, 2020-03-11T22:48:42.637 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:48:42.638 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.638 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:48:42.638 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:48:42.638 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:48:42.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:42.639 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:48:42.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:48:42.640 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:48:42.640 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:48:42.641 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:42.641 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:48:42.641 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:48:42.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:48:42.642 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:42.642 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:48:42.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:48:42.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:48:42.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:48:42.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:42.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:48:42.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:48:42.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:48:42.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:42.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:48:42.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:48:42.651 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:48:42.652 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:48:42.652 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:42.652 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:48:42.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:48:42.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:48:42.655 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:42.656 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:48:42.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:48:42.658 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:48:42.658 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:48:42.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:42.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:48:42.662 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:48:42.663 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:48:42.663 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:48:42.664 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:48:42.664 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:48:42.664 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:48:42.664 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:48:42.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:48:42.751 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:48:42.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:48:42.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:48:42.752 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:48:43.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:48:43.313 INFO:tasks.workunit.client.0.smithi118.stderr:1' 2020-03-11T22:48:43.313 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:48:43.314 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:43.314 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:48:43.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836501 2020-03-11T22:48:43.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836501' 2020-03-11T22:48:43.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:43.443 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:48:43.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672980 2020-03-11T22:48:43.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836501 1-42949672980' 2020-03-11T22:48:43.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:43.570 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836501 2020-03-11T22:48:43.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:43.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:43.572 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836501 2020-03-11T22:48:43.572 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:43.573 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836501 2020-03-11T22:48:43.573 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2020-03-11T22:48:43.573 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836501 2020-03-11T22:48:43.574 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:43.986 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836501 2020-03-11T22:48:43.986 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:44.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:44.988 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:45.347 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836501 2020-03-11T22:48:45.348 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:46.348 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:48:46.349 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:46.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836501 -lt 21474836501 2020-03-11T22:48:46.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:46.701 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672980 2020-03-11T22:48:46.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:46.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:46.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672980 2020-03-11T22:48:46.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:46.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672980 2020-03-11T22:48:46.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2020-03-11T22:48:46.705 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672980 2020-03-11T22:48:46.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:47.165 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672980 -lt 42949672980 2020-03-11T22:48:47.165 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:47.166 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:47.166 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:47.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:48:47.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:47.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:47.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:47.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:47.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:47.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:47.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:48.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:48.019 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:48.019 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:48.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:48.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:48.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:48:48.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:48:48.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:48:48.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:48.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:48.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:48.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:48.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:48.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:48.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:48.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:48.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:48.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:49.049 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:49.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:49.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:49.051 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:49.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:49.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:49.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:49.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:49.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:49.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:49.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:49.602 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:49.602 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:50.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27814501 2020-03-11T22:48:50.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27814501 '!=' null 2020-03-11T22:48:50.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:50.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:50.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:50.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:50.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:50.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:50.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:50.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:50.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:50.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:50.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:50.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:50.525 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:50.525 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:50.525 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:50.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:50.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:50.982 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:50.982 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:51.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27814501 2020-03-11T22:48:51.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27814501 '!=' null 2020-03-11T22:48:51.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:51.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:51.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:51.684 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:51.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:51.686 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:51.687 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:51.687 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:51.687 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:51.687 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:51.687 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:52.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:52.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:52.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:52.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:52.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:52.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:52.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:52.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:52.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:52.617 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:52.617 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:52.617 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:52.617 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:53.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26208970 2020-03-11T22:48:53.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26208970 '!=' null 2020-03-11T22:48:53.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:53.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:53.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:53.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:53.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:53.339 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:53.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:53.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:53.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:53.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:53.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:53.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:53.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:53.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:53.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:54.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:54.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:54.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:54.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:54.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:54.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:54.140 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:54.141 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:54.141 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39313889 2020-03-11T22:48:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39313889 '!=' null 2020-03-11T22:48:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:54.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:54.758 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:54.758 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:54.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:54.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:54.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:54.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:54.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:55.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:55.097 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:55.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:55.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:55.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:55.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:55.654 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:55.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:55.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:55.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:55.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:55.656 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:55.656 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:56.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957025 2020-03-11T22:48:56.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957025 '!=' null 2020-03-11T22:48:56.109 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:56.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:56.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:56.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:56.210 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:56.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:56.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:56.211 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:56.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:56.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:56.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:56.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:56.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:56.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:56.553 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:57.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:57.005 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:57.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:57.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:57.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957025 2020-03-11T22:48:57.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957025 '!=' null 2020-03-11T22:48:57.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:57.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:57.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:57.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:57.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:57.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:57.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:57.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:57.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:57.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:57.557 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:57.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:57.895 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:57.896 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:57.896 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:58.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:58.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:58.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:58.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:58.983 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325506 2020-03-11T22:48:58.983 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325506 '!=' null 2020-03-11T22:48:58.983 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:58.983 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:58.984 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:59.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:59.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:59.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:59.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:59.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:59.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:59.422 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:59.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:59.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:59.996 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:59.996 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:00.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=35820260 2020-03-11T22:49:00.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 35820260 '!=' null 2020-03-11T22:49:00.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:00.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:00.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:00.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:00.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:00.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:00.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:00.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:00.645 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:00.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:00.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:01.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:01.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:01.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:01.038 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:01.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:01.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:01.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:01.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:01.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:01.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:01.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:01.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:01.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:02.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831187 2020-03-11T22:49:02.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831187 '!=' null 2020-03-11T22:49:02.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:02.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:02.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:02.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:02.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:02.195 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:02.195 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:02.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:02.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:02.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:02.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:02.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:02.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:02.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:02.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:03.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:03.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:03.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:03.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:03.017 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:03.017 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:03.017 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:03.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:03.018 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:03.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831187 2020-03-11T22:49:03.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831187 '!=' null 2020-03-11T22:49:03.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:03.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:03.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:03.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:03.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:04.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:04.062 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:04.063 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:04.063 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:04.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:04.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:04.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:04.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:04.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:04.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:04.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:04.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:04.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:05.265 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=42809977 2020-03-11T22:49:05.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 42809977 '!=' null 2020-03-11T22:49:05.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:05.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:05.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:05.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:05.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:05.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:05.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:05.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:05.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:05.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:05.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:05.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:05.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:05.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:05.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:06.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:06.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:06.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:06.226 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:06.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704869 2020-03-11T22:49:06.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704869 '!=' null 2020-03-11T22:49:06.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:06.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:06.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:06.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:06.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:06.788 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:06.788 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:06.788 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:06.788 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:06.788 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:06.789 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:07.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:07.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:07.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:07.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:07.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:07.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:07.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:08.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36694387 2020-03-11T22:49:08.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36694387 '!=' null 2020-03-11T22:49:08.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:08.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:08.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:08.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:08.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:08.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:08.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:08.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:08.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:08.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:08.285 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:08.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:08.620 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:08.620 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:08.620 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:09.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:09.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:09.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:09.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:09.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:09.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:09.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:09.094 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:09.094 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:09.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957440 2020-03-11T22:49:09.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957440 '!=' null 2020-03-11T22:49:09.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:09.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:09.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:09.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:09.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:09.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:09.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:09.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:09.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:09.721 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:09.722 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:10.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:10.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:10.054 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:10.054 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:10.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:10.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:10.499 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:10.499 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:10.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957440 2020-03-11T22:49:10.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957440 '!=' null 2020-03-11T22:49:10.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:10.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:10.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:11.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:11.048 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:11.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:11.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:11.049 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:11.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:11.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:11.050 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:11.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:11.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:11.385 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:11.385 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:11.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:11.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:11.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:11.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:11.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:11.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:11.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:11.830 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:11.830 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:12.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20966665 2020-03-11T22:49:12.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20966665 '!=' null 2020-03-11T22:49:12.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:12.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:12.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:12.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:12.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:12.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:12.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:12.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:12.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:12.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:12.383 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:12.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:12.712 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:12.713 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:12.713 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:13.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:13.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:13.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:13.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:13.156 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:13.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:13.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11670982 2020-03-11T22:49:13.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11670982 '!=' null 2020-03-11T22:49:13.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:13.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:13.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:13.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:13.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:13.704 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:13.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:13.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:13.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:13.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:13.705 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:14.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:49:14.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:14.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:14.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:14.484 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:49:14.484 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:14.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:14.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:49:14.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:49:14.485 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:49:14.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:49:14.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:49:14.486 INFO:tasks.workunit.client.0.smithi118.stdout:TEST PASSED 2020-03-11T22:49:14.991 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test1' does not exist 2020-03-11T22:49:14.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:49:15.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:49:15.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:49:15.542 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test2' does not exist 2020-03-11T22:49:15.550 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:49:15.551 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:15.551 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:15.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:15.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:15.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:15.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:15.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:15.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:15.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:15.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:15.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:15.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:15.670 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:15.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:15.672 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:15.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:15.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:15.672 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55883 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:15.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:15.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:15.694 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:15.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:15.695 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:15.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:15.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:15.695 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:15.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:15.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:15.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:15.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:15.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:15.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:15.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:49:15.698 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:15.698 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:15.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:15.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55883 2020-03-11T22:49:15.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:15.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:15.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:49:15.701 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:49:15.701 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:15.701 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:15.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.55883 2020-03-11T22:49:15.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:49:15.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:49:15.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:49:15.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:49:15.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:49:15.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:49:15.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:49:15.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:49:15.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:49:15.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:49:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:49:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:49:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:49:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:49:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:49:15.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:49:15.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:49:15.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:49:15.950 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:49:15.950 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:15.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:15.951 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:15.951 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:15.951 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:15.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:15.952 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:49:16.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:49:16.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:49:16.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:16.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:16.100 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:16.101 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:16.102 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:16.102 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-mon.a.asok 2020-03-11T22:49:16.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:16.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55883/ceph-mon.a.asok config get fsid 2020-03-11T22:49:16.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:49:16.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:16.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:16.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:49:16.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:49:16.188 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:16.188 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:16.188 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:16.188 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:16.188 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:16.189 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:16.189 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-mon.a.asok 2020-03-11T22:49:16.189 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:16.189 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55883/ceph-mon.a.asok config get mon_host 2020-03-11T22:49:16.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:49:16.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:49:16.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:49:16.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:49:16.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:49:16.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:49:16.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:49:17.174 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:17.175 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:17.176 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:17.176 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:17.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:49:17.201 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:49:17.212 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:49:17.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:17.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:49:17.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:17.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:17.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:49:17.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:17.214 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:49:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:17.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:49:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:49:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:17.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:17.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:17.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:17.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:17.217 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:17.218 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:17.218 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:17.218 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:17.218 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:17.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:17.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:17.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:17.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ad10b14b-8026-4a72-9309-ef94d3245d7d 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 ad10b14b-8026-4a72-9309-ef94d3245d7d' 2020-03-11T22:49:17.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:17.222 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 ad10b14b-8026-4a72-9309-ef94d3245d7d 2020-03-11T22:49:17.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDtamlenQMTDRAAogUIyfiQpJoQlrfVESApQw== 2020-03-11T22:49:17.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDtamlenQMTDRAAogUIyfiQpJoQlrfVESApQw=="}' 2020-03-11T22:49:17.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ad10b14b-8026-4a72-9309-ef94d3245d7d -i td/osd-backfill-prio/0/new.json 2020-03-11T22:49:17.597 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:17.607 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:49:17.607 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDtamlenQMTDRAAogUIyfiQpJoQlrfVESApQw== --osd-uuid ad10b14b-8026-4a72-9309-ef94d3245d7d 2020-03-11T22:49:17.633 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.630+0000 7f59e62cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:17.634 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.632+0000 7f59e62cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:17.637 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.634+0000 7f59e62cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:17.637 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.634+0000 7f59e62cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:17.637 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.634+0000 7f59e62cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:17.638 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.634+0000 7f59e62cbec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:17.681 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:17.678+0000 7f59e62cbec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:49:20.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:49:20.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:20.634 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:49:20.634 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:20.634 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:49:21.085 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:49:21.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:49:21.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:21.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:21.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:21.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:21.108 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:21.106+0000 7f90ef290ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:21.124 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:21.122+0000 7f90ef290ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:21.126 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:21.124+0000 7f90ef290ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:21.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:49:21.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:21.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:21.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:21.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:21.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:21.434 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:21.434 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:21.434 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:21.434 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:21.641 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:21.638+0000 7f90ef290ec0 -1 Falling back to public interface 2020-03-11T22:49:21.789 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:22.381 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:22.378+0000 7f90ef290ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:49:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:22.792 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:22.792 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:23.148 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/62775,v1:127.0.0.1:6803/62775] [v2:127.0.0.1:6804/62775,v1:127.0.0.1:6805/62775] exists,up ad10b14b-8026-4a72-9309-ef94d3245d7d 2020-03-11T22:49:23.149 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 009f896b-3261-4fe9-a1e9-3a5b3b54506c 2020-03-11T22:49:23.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:23.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:23.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:23.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:49:23.150 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:23.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:23.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:23.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:23.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:23.152 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:23.152 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:23.153 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:23.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:49:23.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:23.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=009f896b-3261-4fe9-a1e9-3a5b3b54506c 2020-03-11T22:49:23.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 009f896b-3261-4fe9-a1e9-3a5b3b54506c' 2020-03-11T22:49:23.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:23.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDzamleWwSkCRAAq0pCdXA6eS2RGzQSRbaGbg== 2020-03-11T22:49:23.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDzamleWwSkCRAAq0pCdXA6eS2RGzQSRbaGbg=="}' 2020-03-11T22:49:23.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 009f896b-3261-4fe9-a1e9-3a5b3b54506c -i td/osd-backfill-prio/1/new.json 2020-03-11T22:49:23.718 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:23.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDzamleWwSkCRAAq0pCdXA6eS2RGzQSRbaGbg== --osd-uuid 009f896b-3261-4fe9-a1e9-3a5b3b54506c 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.752+0000 7f2102558ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.761+0000 7f2102558ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.762+0000 7f2102558ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.763+0000 7f2102558ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:23.851 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.763+0000 7f2102558ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:23.852 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.763+0000 7f2102558ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:23.897 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:23.894+0000 7f2102558ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:49:26.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:49:26.915 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:26.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:49:26.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:26.916 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:49:27.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:49:27.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:27.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:27.369 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:27.369 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:27.369 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:49:27.392 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:27.389+0000 7f3f4c29cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.393 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:27.391+0000 7f3f4c29cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.396 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:27.392+0000 7f3f4c29cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:49:27.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:27.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:27.714 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:27.715 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:27.912 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:27.909+0000 7f3f4c29cec0 -1 Falling back to public interface 2020-03-11T22:49:28.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:28.722 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:28.719+0000 7f3f4c29cec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:49:29.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:29.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:29.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:29.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:29.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:29.068 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:29.426 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63130,v1:127.0.0.1:6811/63130] [v2:127.0.0.1:6812/63130,v1:127.0.0.1:6813/63130] exists,up 009f896b-3261-4fe9-a1e9-3a5b3b54506c 2020-03-11T22:49:29.426 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 d2cdd230-099c-41fe-acba-74ff1d513a9a 2020-03-11T22:49:29.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:29.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:49:29.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:29.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:49:29.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:29.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:29.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:29.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:29.429 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:29.430 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:29.431 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:29.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d2cdd230-099c-41fe-acba-74ff1d513a9a 2020-03-11T22:49:29.433 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d2cdd230-099c-41fe-acba-74ff1d513a9a' 2020-03-11T22:49:29.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:29.441 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD5amletCskGhAAor93gzDQ6UQVIxrRD1WPjA== 2020-03-11T22:49:29.441 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD5amletCskGhAAor93gzDQ6UQVIxrRD1WPjA=="}' 2020-03-11T22:49:29.441 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d2cdd230-099c-41fe-acba-74ff1d513a9a -i td/osd-backfill-prio/2/new.json 2020-03-11T22:49:29.867 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:49:29.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:49:29.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD5amletCskGhAAor93gzDQ6UQVIxrRD1WPjA== --osd-uuid d2cdd230-099c-41fe-acba-74ff1d513a9a 2020-03-11T22:49:29.904 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.900+0000 7fbd34508ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.904 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.902+0000 7fbd34508ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.907 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.903+0000 7fbd34508ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.907 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.904+0000 7fbd34508ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:29.907 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.904+0000 7fbd34508ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:29.908 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.904+0000 7fbd34508ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:29.962 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:29.959+0000 7fbd34508ec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:49:32.936 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:49:32.936 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:32.937 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:49:32.937 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:49:32.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:33.402 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:49:33.402 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:33.402 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:33.402 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:33.403 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:33.403 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:49:33.425 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:33.422+0000 7ff678e86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.427 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:33.424+0000 7ff678e86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.427 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:33.425+0000 7ff678e86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:33.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:33.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:33.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:33.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:33.750 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:33.947 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:33.944+0000 7ff678e86ec0 -1 Falling back to public interface 2020-03-11T22:49:34.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:34.698 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:34.695+0000 7ff678e86ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:49:35.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:35.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:35.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:35.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:35.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:35.101 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:35.478 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63485,v1:127.0.0.1:6819/63485] [v2:127.0.0.1:6820/63485,v1:127.0.0.1:6821/63485] exists,up d2cdd230-099c-41fe-acba-74ff1d513a9a 2020-03-11T22:49:35.478 INFO:tasks.workunit.client.0.smithi118.stdout:add osd3 351fd94d-fc4b-42e3-91b8-defe6fd4289b 2020-03-11T22:49:35.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:49:35.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:35.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:35.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:49:35.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:35.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:49:35.481 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:35.481 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:35.481 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:35.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:35.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:49:35.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:49:35.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:35.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:35.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:35.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:35.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:35.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:35.485 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:35.485 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:35.486 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:35.486 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:35.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:35.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:35.487 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:35.488 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:49:35.489 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:35.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=351fd94d-fc4b-42e3-91b8-defe6fd4289b 2020-03-11T22:49:35.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 351fd94d-fc4b-42e3-91b8-defe6fd4289b' 2020-03-11T22:49:35.489 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:35.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD/amle3A4XHRAA5XwNvt0PhM+z8koH0K4NLw== 2020-03-11T22:49:35.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD/amle3A4XHRAA5XwNvt0PhM+z8koH0K4NLw=="}' 2020-03-11T22:49:35.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 351fd94d-fc4b-42e3-91b8-defe6fd4289b -i td/osd-backfill-prio/3/new.json 2020-03-11T22:49:36.037 INFO:tasks.workunit.client.0.smithi118.stdout:3 2020-03-11T22:49:36.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:49:36.047 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD/amle3A4XHRAA5XwNvt0PhM+z8koH0K4NLw== --osd-uuid 351fd94d-fc4b-42e3-91b8-defe6fd4289b 2020-03-11T22:49:36.072 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.070+0000 7f5120115ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.121 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.118+0000 7f5120115ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.122 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.119+0000 7f5120115ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.122 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.120+0000 7f5120115ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:36.123 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.120+0000 7f5120115ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:36.123 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.120+0000 7f5120115ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:36.177+0000 7f5120115ec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:49:39.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:49:39.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:39.300 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd3 key to auth repository 2020-03-11T22:49:39.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:49:39.300 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:39.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:49:39.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:39.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:39.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:39.756 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.3 2020-03-11T22:49:39.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:39.779 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:39.777+0000 7f944e8c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:39.781 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:39.778+0000 7f944e8c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:39.782 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:39.780+0000 7f944e8c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:40.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:40.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:40.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:40.102 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:40.297 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:40.294+0000 7f944e8c4ec0 -1 Falling back to public interface 2020-03-11T22:49:40.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:41.047 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:41.044+0000 7f944e8c4ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:49:41.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:41.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:41.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:41.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:41.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:41.453 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:41.825 INFO:tasks.workunit.client.0.smithi118.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/63840,v1:127.0.0.1:6827/63840] [v2:127.0.0.1:6828/63840,v1:127.0.0.1:6829/63840] exists,up 351fd94d-fc4b-42e3-91b8-defe6fd4289b 2020-03-11T22:49:41.825 INFO:tasks.workunit.client.0.smithi118.stdout:add osd4 3cc859ca-f1cc-4247-81da-0002c468c2c1 2020-03-11T22:49:41.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:41.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:41.825 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:49:41.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:41.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:49:41.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:41.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:41.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:41.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:41.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' 2020-03-11T22:49:41.829 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:41.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3cc859ca-f1cc-4247-81da-0002c468c2c1 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 3cc859ca-f1cc-4247-81da-0002c468c2c1' 2020-03-11T22:49:41.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:41.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAFa2lefGsGMhAAPGHhLTEqlOZizYt1P1xhyg== 2020-03-11T22:49:41.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAFa2lefGsGMhAAPGHhLTEqlOZizYt1P1xhyg=="}' 2020-03-11T22:49:41.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3cc859ca-f1cc-4247-81da-0002c468c2c1 -i td/osd-backfill-prio/4/new.json 2020-03-11T22:49:42.368 INFO:tasks.workunit.client.0.smithi118.stdout:4 2020-03-11T22:49:42.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:49:42.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAFa2lefGsGMhAAPGHhLTEqlOZizYt1P1xhyg== --osd-uuid 3cc859ca-f1cc-4247-81da-0002c468c2c1 2020-03-11T22:49:42.403 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.401+0000 7fad5cd89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.447 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.444+0000 7fad5cd89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.449 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.446+0000 7fad5cd89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.450 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.447+0000 7fad5cd89ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:42.450 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.447+0000 7fad5cd89ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:42.450 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.447+0000 7fad5cd89ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:42.522 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:42.519+0000 7fad5cd89ec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:49:45.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:49:45.783 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:45.784 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd4 key to auth repository 2020-03-11T22:49:45.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:49:45.784 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:46.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:49:46.244 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.4 2020-03-11T22:49:46.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=41a1c503-2129-4311-b925-f0b72490b1fe --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55883/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:46.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:46.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:46.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:46.268 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:46.264+0000 7f39f8513ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.283 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:46.279+0000 7f39f8513ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.284 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:46.280+0000 7f39f8513ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:46.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:46.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:46.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:46.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:49:46.602 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:49:46.799 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:46.796+0000 7f39f8513ec0 -1 Falling back to public interface 2020-03-11T22:49:46.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:47.548 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:49:47.544+0000 7f39f8513ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:49:47.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:47.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:47.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:47.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:47.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:49:47.964 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:49:48.407 INFO:tasks.workunit.client.0.smithi118.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64203,v1:127.0.0.1:6835/64203] [v2:127.0.0.1:6836/64203,v1:127.0.0.1:6837/64203] exists,up 3cc859ca-f1cc-4247-81da-0002c468c2c1 2020-03-11T22:49:48.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:48.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:48.408 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:48.408 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:48.408 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:48.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:49:48.409 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:49:48.839 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test1' already exists 2020-03-11T22:49:48.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:49.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:49:50.513 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 1 size to 2 2020-03-11T22:49:50.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:50.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:49:50.524 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:49:51.237 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test2' already exists 2020-03-11T22:49:51.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:52.246 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:49:52.894 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 2 size to 2 2020-03-11T22:49:52.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:52.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:49:52.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:49:53.408 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test3' already exists 2020-03-11T22:49:53.415 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:49:55.217 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 3 size to 2 2020-03-11T22:49:55.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:55.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:49:55.226 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:49:55.716 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test4' already exists 2020-03-11T22:49:55.723 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:56.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:49:57.442 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 4 size to 2 2020-03-11T22:49:57.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:57.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:49:57.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:49:58.253 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test5' already exists 2020-03-11T22:49:58.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:59.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:50:00.084 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 5 size to 2 2020-03-11T22:50:00.090 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:00.090 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:50:00.090 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:50:00.589 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test6' already exists 2020-03-11T22:50:00.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:01.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:50:02.247 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 6 size to 2 2020-03-11T22:50:02.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:02.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:50:02.255 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:50:03.008 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test7' already exists 2020-03-11T22:50:03.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:04.018 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:50:04.694 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 7 size to 2 2020-03-11T22:50:04.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:04.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:50:04.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:50:05.229 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test8' already exists 2020-03-11T22:50:05.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:06.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:50:07.070 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 8 size to 2 2020-03-11T22:50:07.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:07.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:50:07.078 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:50:07.560 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test9' already exists 2020-03-11T22:50:07.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:08.569 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:50:09.252 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 9 size to 2 2020-03-11T22:50:09.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:09.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:50:09.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:50:09.860 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test10' already exists 2020-03-11T22:50:09.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:10.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:50:11.510 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 10 size to 2 2020-03-11T22:50:11.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:50:16.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:50:16.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:16.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:16.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:16.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:16.520 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:16.521 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:16.521 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:16.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:16.521 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:16.522 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:16.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:16.625 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:16.625 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:16.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:16.969 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:16.970 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:17.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:50:17.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:50:17.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:17.095 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:50:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:50:17.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:17.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:17.346 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:50:17.347 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-60129542155' 2020-03-11T22:50:17.347 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:17.347 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:17.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378634 2020-03-11T22:50:17.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-60129542155 3-81604378634' 2020-03-11T22:50:17.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:17.473 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:17.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215112 2020-03-11T22:50:17.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-60129542155 3-81604378634 4-103079215112' 2020-03-11T22:50:17.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:17.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:50:17.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:17.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:17.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:50:17.600 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:17.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:50:17.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:50:17.601 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:17.601 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:50:17.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:50:17.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:18.943 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:18.943 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:19.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:50:19.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:19.297 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:50:19.297 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:19.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:19.298 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:50:19.298 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:19.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:50:19.299 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:50:19.299 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:19.299 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:50:19.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:50:19.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:19.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:50:19.643 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:19.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:19.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:50:19.644 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:19.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:50:19.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:50:19.646 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:19.646 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:50:19.991 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:50:19.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:19.992 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378634 2020-03-11T22:50:19.992 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:19.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:19.993 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378634 2020-03-11T22:50:19.993 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:19.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378634 2020-03-11T22:50:19.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2020-03-11T22:50:19.994 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:19.995 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378634 2020-03-11T22:50:20.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378634 -lt 81604378634 2020-03-11T22:50:20.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:20.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215112 2020-03-11T22:50:20.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:20.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:20.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215112 2020-03-11T22:50:20.343 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:20.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215112 2020-03-11T22:50:20.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215112' 2020-03-11T22:50:20.344 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:20.344 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215112 2020-03-11T22:50:20.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215113 -lt 103079215112 2020-03-11T22:50:20.792 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:20.792 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:20.792 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:50:21.256 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:21.256 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:21.256 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:21.256 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:21.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:21.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:21.257 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:21.606 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:50:21.606 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:21.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:21.607 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:22.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:50:22.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:22.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:22.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:22.381 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:22.381 INFO:tasks.workunit.client.0.smithi118.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:11.820509+0000 0'0 81:20 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:50:09.640516+0000 0'0 2020-03-11T22:50:09.640516+0000 0 2020-03-11T22:50:22.381 INFO:tasks.workunit.client.0.smithi118.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:09.968023+0000 0'0 80:25 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:50:07.386092+0000 0'0 2020-03-11T22:50:07.386092+0000 0 2020-03-11T22:50:22.381 INFO:tasks.workunit.client.0.smithi118.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:07.773368+0000 0'0 78:28 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:50:05.044620+0000 0'0 2020-03-11T22:50:05.044620+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:05.408471+0000 0'0 80:35 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:50:02.558711+0000 0'0 2020-03-11T22:50:02.558711+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:53.722507+0000 0'0 78:44 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:49:50.955054+0000 0'0 2020-03-11T22:49:50.955054+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:50.872133+0000 0'0 80:56 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:48.661846+0000 0'0 2020-03-11T22:49:48.661846+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:55.514925+0000 0'0 80:50 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:53.206251+0000 0'0 2020-03-11T22:49:53.206251+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:58.114771+0000 0'0 79:46 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:49:55.528678+0000 0'0 2020-03-11T22:49:55.528678+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:00.341334+0000 0'0 78:39 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:49:57.829261+0000 0'0 2020-03-11T22:49:57.829261+0000 0 2020-03-11T22:50:22.382 INFO:tasks.workunit.client.0.smithi118.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:03.791935+0000 0'0 79:35 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:50:00.387943+0000 0'0 2020-03-11T22:50:00.387943+0000 0 2020-03-11T22:50:22.383 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:50:22.383 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:22.383 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:50:22.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:50:22.393 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:22.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:22.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:50:22.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:22.775 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:22.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:22.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:22.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:50:22.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:50:22.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:22.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:50:22.779 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:23.113 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:50:23.114 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:23.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:50:23.116 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:23.475 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.476 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:23.476 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.477 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:50:23.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:23.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:50:23.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:23.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.818 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:50:23.818 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:23.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:50:23.819 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:23.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:50:23.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:50:23.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:23.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:50:23.820 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:24.156 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:50:24.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:50:24.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:24.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:50:24.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:50:24.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:24.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:50:24.159 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:24.537 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:50:24.538 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:50:24.540 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:24.875 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:24.876 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:24.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:50:24.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:50:24.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:50:24.880 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:50:24.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:50:24.881 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:50:25.389 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test2' does not exist 2020-03-11T22:50:25.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:25.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:50:25.397 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:25.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:50:25.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:50:25.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:50:25.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:50:25.884 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test4' does not exist 2020-03-11T22:50:25.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:25.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:50:25.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:50:25.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:50:25.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:50:26.399 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test5' does not exist 2020-03-11T22:50:26.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:26.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:50:26.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:50:26.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:50:26.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:50:27.029 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test6' does not exist 2020-03-11T22:50:27.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:27.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:50:27.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:27.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:50:27.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:50:27.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:50:27.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:50:27.545 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test8' does not exist 2020-03-11T22:50:27.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:27.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:50:27.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:50:27.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:50:27.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:50:28.065 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test9' does not exist 2020-03-11T22:50:28.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:28.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:50:28.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:50:28.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:50:28.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:50:28.614 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test10' does not exist 2020-03-11T22:50:28.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:50:29.404 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 3 size to 1 2020-03-11T22:50:29.412 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:50:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 7 size to 1 2020-03-11T22:50:30.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:50:30.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:30.366 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:30.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:30.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:30.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:30.368 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:30.368 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:30.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:30.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:30.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:30.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:30.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:30.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:30.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:30.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:30.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:30.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:30.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:30.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:50:30.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:50:30.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:30.934 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:31.059 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672977 2020-03-11T22:50:31.059 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2020-03-11T22:50:31.060 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:31.060 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:31.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542160 2020-03-11T22:50:31.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542160' 2020-03-11T22:50:31.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:31.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:31.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378638 2020-03-11T22:50:31.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542160 3-81604378638' 2020-03-11T22:50:31.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:31.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:31.435 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215117 2020-03-11T22:50:31.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542160 3-81604378638 4-103079215117' 2020-03-11T22:50:31.436 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:31.436 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:50:31.436 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:31.437 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:31.437 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:50:31.437 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:31.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:50:31.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:50:31.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:31.438 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:50:31.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836498 2020-03-11T22:50:31.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:32.857 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:32.858 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836498 2020-03-11T22:50:33.220 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672977 2020-03-11T22:50:33.220 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:33.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672977 2020-03-11T22:50:33.221 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672977 2020-03-11T22:50:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-03-11T22:50:33.223 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:33.223 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672977 2020-03-11T22:50:33.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672977 -lt 42949672977 2020-03-11T22:50:33.570 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.570 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542160 2020-03-11T22:50:33.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:33.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542160 2020-03-11T22:50:33.572 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.572 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542160 2020-03-11T22:50:33.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542160 2020-03-11T22:50:33.573 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2020-03-11T22:50:33.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:33.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542160 -lt 60129542160 2020-03-11T22:50:33.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378638 2020-03-11T22:50:33.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:33.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378638 2020-03-11T22:50:33.925 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378638 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378638 2020-03-11T22:50:34.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378639 -lt 81604378638 2020-03-11T22:50:34.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:34.273 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215117 2020-03-11T22:50:34.273 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:34.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:34.274 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215117 2020-03-11T22:50:34.274 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:34.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215117 2020-03-11T22:50:34.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215117' 2020-03-11T22:50:34.276 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:34.276 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215117 2020-03-11T22:50:34.682 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215118 -lt 103079215117 2020-03-11T22:50:34.682 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:34.683 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:34.683 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:35.127 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:50:35.127 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:35.127 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:35.128 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:35.128 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:35.128 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:35.128 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:35.128 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:35.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:50:35.462 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:35.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:35.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:35.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:50:35.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:35.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:35.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:50:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:10+0 records in 2020-03-11T22:50:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:10+0 records out 2020-03-11T22:50:35.955 INFO:tasks.workunit.client.0.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0456569 s, 230 MB/s 2020-03-11T22:50:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:50:35.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:50:35.956 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:50:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:35.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:50:36.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:36.407 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:50:36.874 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:36.875 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:50:37.369 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:37.370 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:50:37.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:37.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:50:38.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:38.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:50:38.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:38.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:50:39.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:39.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:50:39.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:39.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:50:40.177 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:40.178 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:50:40.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:40.637 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:50:41.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:41.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:50:41.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:41.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:50:42.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:42.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:50:42.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:42.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:50:42.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:42.967 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:50:43.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:43.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:50:43.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:43.951 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:50:44.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:44.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.009 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.959 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.416 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.876 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.427 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.802 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.194 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.882 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.833 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.834 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.303 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.821 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.821 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.244 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.245 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.734 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.212 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.670 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.671 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:50:57.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:50:57.640 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.640 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:50:58.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:50:58.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:50:59.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:50:59.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:51:00.081 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:51:00.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:51:00.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:51:00.082 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:51:00.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.355 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.659 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.918 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.439 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.773 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.815 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.110 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.111 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.605 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.879 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.421 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.422 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:51:09.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:51:09.347 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.348 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:51:09.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:51:09.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:51:10.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:51:10.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:51:10.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:51:10.965 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.965 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:51:11.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:51:11.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:51:11.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.732 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:51:11.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.982 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:51:12.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:51:12.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:51:12.897 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:51:13.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:51:13.425 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.425 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:51:13.658 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:51:13.659 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:51:13.659 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:51:13.659 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:51:13.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.660 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.925 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.442 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.772 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.908 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.909 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.027 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.365 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:51:21.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:21.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:51:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:51:21.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:21.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:51:22.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:22.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:51:22.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:22.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:51:22.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:22.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:51:23.141 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:23.142 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:51:23.387 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:23.388 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:51:23.684 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:23.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:51:23.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:23.946 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:51:24.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:24.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:51:24.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:24.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:51:24.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:24.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:51:25.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:25.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:51:25.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:25.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:51:25.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:25.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:51:25.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:25.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:51:26.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:26.089 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:51:26.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:26.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:51:26.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:26.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:51:26.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:26.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:51:27.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:27.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:51:27.353 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:51:27.354 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:51:27.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:51:27.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:51:27.355 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:51:27.355 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:51:27.355 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:51:27.356 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:51:27.356 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:27.356 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:51:27.715 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:51:27.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:51:27.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:28.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:51:28.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:28.394 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:28.394 INFO:tasks.workunit.client.0.smithi118.stdout:7.0 41 0 0 0 0 429916160 0 0 23 23 active+clean 2020-03-11T22:50:30.220420+0000 94'123 94:176 [1] 1 [1] 1 0'0 2020-03-11T22:50:02.558711+0000 0'0 2020-03-11T22:50:02.558711+0000 0 2020-03-11T22:51:28.394 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:50.872133+0000 94'150 94:217 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:48.661846+0000 0'0 2020-03-11T22:49:48.661846+0000 0 2020-03-11T22:51:28.394 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:29.859350+0000 94'150 94:218 [1] 1 [1] 1 0'0 2020-03-11T22:49:53.206251+0000 0'0 2020-03-11T22:49:53.206251+0000 0 2020-03-11T22:51:28.395 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:51:28.395 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:28.397 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:51:28.403 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:51:28.404 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:51:29.106 INFO:tasks.workunit.client.0.smithi118.stderr:nobackfill is set 2020-03-11T22:51:29.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:51:29.799 INFO:tasks.workunit.client.0.smithi118.stderr:noout is set 2020-03-11T22:51:29.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:51:30.418 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 7 size to 2 2020-03-11T22:51:30.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:51:32.428 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:32.428 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:32.428 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:32.429 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:32.429 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:32.429 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:51:32.429 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:51:32.429 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:32.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:32.528 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:51:32.528 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:51:32.528 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:32.528 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:32.529 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:32.530 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:51:32.531 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:32.531 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:51:32.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:51:32.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:32.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:32.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:32.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:51:32.871 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:51:34.872 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:35.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 2 = 10 ']' 2020-03-11T22:51:35.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:51:37.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:37.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:37.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:37.567 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:51:37.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:37.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:37.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:37.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:38.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836513 2020-03-11T22:51:38.061 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513' 2020-03-11T22:51:38.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:38.062 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:38.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672992 2020-03-11T22:51:38.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992' 2020-03-11T22:51:38.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:38.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:38.311 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542175 2020-03-11T22:51:38.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-60129542175' 2020-03-11T22:51:38.312 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:38.312 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:38.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378654 2020-03-11T22:51:38.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-60129542175 3-81604378654' 2020-03-11T22:51:38.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:38.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215132 2020-03-11T22:51:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-60129542175 3-81604378654 4-103079215132' 2020-03-11T22:51:38.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:38.564 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836513 2020-03-11T22:51:38.564 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:38.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:38.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836513 2020-03-11T22:51:38.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:38.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836513 2020-03-11T22:51:38.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2020-03-11T22:51:38.566 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:38.566 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836513 2020-03-11T22:51:38.914 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836513 -lt 21474836513 2020-03-11T22:51:38.914 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:38.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672992 2020-03-11T22:51:38.915 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:38.916 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:38.916 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672992 2020-03-11T22:51:38.916 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:38.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672992 2020-03-11T22:51:38.917 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2020-03-11T22:51:38.917 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:38.918 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672992 2020-03-11T22:51:39.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672992 -lt 42949672992 2020-03-11T22:51:39.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:39.275 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542175 2020-03-11T22:51:39.275 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:39.276 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:39.276 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542175 2020-03-11T22:51:39.276 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:39.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542175 2020-03-11T22:51:39.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542175' 2020-03-11T22:51:39.278 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:39.278 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542175 2020-03-11T22:51:39.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542175 -lt 60129542175 2020-03-11T22:51:39.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:39.630 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378654 2020-03-11T22:51:39.631 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:39.631 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:39.632 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378654 2020-03-11T22:51:39.632 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:39.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378654 2020-03-11T22:51:39.633 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378654' 2020-03-11T22:51:39.633 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:39.633 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378654 2020-03-11T22:51:39.977 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378654 -lt 81604378654 2020-03-11T22:51:39.977 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:39.978 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215132 2020-03-11T22:51:39.978 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:39.978 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:39.979 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215132 2020-03-11T22:51:39.979 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:39.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215132 2020-03-11T22:51:39.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215132' 2020-03-11T22:51:39.980 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215132 2020-03-11T22:51:39.981 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:40.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215130 -lt 103079215132 2020-03-11T22:51:40.325 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:41.326 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:41.326 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215133 -lt 103079215132 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:41.684 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:41.685 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:51:41.685 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:51:41.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:41.685 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:41.771 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:51:41.771 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:51:41.771 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:51:41.772 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:41.773 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:41.774 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:51:41.774 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:41.774 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:51:41.778 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:51:42.263 INFO:tasks.workunit.client.0.smithi118.stderr:osd.0 is already out. 2020-03-11T22:51:42.270 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:51:44.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:44.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:44.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:44.630 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:44.630 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:51:44.630 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:51:44.631 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:51:44.631 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:51:44.631 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:44.631 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:44.631 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:44.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836517 2020-03-11T22:51:44.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517' 2020-03-11T22:51:44.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:44.756 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:44.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672995 2020-03-11T22:51:44.883 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995' 2020-03-11T22:51:44.884 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:44.884 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:45.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542178 2020-03-11T22:51:45.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-60129542178' 2020-03-11T22:51:45.011 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:45.011 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:45.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378657 2020-03-11T22:51:45.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-60129542178 3-81604378657' 2020-03-11T22:51:45.136 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:45.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:45.260 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215136 2020-03-11T22:51:45.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-60129542178 3-81604378657 4-103079215136' 2020-03-11T22:51:45.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:45.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836517 2020-03-11T22:51:45.261 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:45.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:45.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836517 2020-03-11T22:51:45.262 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:45.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836517 2020-03-11T22:51:45.263 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2020-03-11T22:51:45.263 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:45.263 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836517 2020-03-11T22:51:45.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836517 2020-03-11T22:51:45.623 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:46.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:46.624 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:46.978 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836517 -lt 21474836517 2020-03-11T22:51:46.978 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:46.979 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672995 2020-03-11T22:51:46.979 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:46.979 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:46.980 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672995 2020-03-11T22:51:46.980 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:46.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672995 2020-03-11T22:51:46.981 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2020-03-11T22:51:46.981 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:46.981 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672995 2020-03-11T22:51:47.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672996 -lt 42949672995 2020-03-11T22:51:47.328 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:47.328 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542178 2020-03-11T22:51:47.328 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:47.329 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:47.329 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542178 2020-03-11T22:51:47.330 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:47.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542178 2020-03-11T22:51:47.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542178' 2020-03-11T22:51:47.331 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:47.331 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542178 2020-03-11T22:51:47.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542179 -lt 60129542178 2020-03-11T22:51:47.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:47.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378657 2020-03-11T22:51:47.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:47.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:47.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378657 2020-03-11T22:51:47.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:47.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378657 2020-03-11T22:51:47.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378657' 2020-03-11T22:51:47.694 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:47.694 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378657 2020-03-11T22:51:48.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378657 -lt 81604378657 2020-03-11T22:51:48.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:48.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215136 2020-03-11T22:51:48.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:48.040 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:48.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215136 2020-03-11T22:51:48.041 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:48.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215136 2020-03-11T22:51:48.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215136' 2020-03-11T22:51:48.042 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:48.042 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215136 2020-03-11T22:51:48.386 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215136 -lt 103079215136 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.388 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:51:48.389 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:51:48.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:48.389 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:48.476 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:51:48.477 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:51:48.477 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:48.477 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:51:48.478 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:51:48.479 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:48.480 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:48.481 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:51:48.481 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:48.481 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:48.820 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:48.820 INFO:tasks.workunit.client.0.smithi118.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:51:37.462573+0000 94'150 104:221 [1,4] 1 [1] 1 0'0 2020-03-11T22:50:02.558711+0000 0'0 2020-03-11T22:50:02.558711+0000 0 2020-03-11T22:51:48.821 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:51:44.792340+0000 94'150 105:233 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:49:48.661846+0000 0'0 2020-03-11T22:49:48.661846+0000 0 2020-03-11T22:51:48.821 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:29.859350+0000 94'150 104:226 [1] 1 [1] 1 0'0 2020-03-11T22:49:53.206251+0000 0'0 2020-03-11T22:49:53.206251+0000 0 2020-03-11T22:51:48.821 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:51:48.821 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:48.822 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:51:48.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:51:49.454 INFO:tasks.workunit.client.0.smithi118.stderr:set pool 3 size to 2 2020-03-11T22:51:49.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:51:51.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:51.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:51.464 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:51:51.816 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:51.817 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:51.817 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:51.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836520 2020-03-11T22:51:51.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520' 2020-03-11T22:51:51.941 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:51.942 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:52.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672999 2020-03-11T22:51:52.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999' 2020-03-11T22:51:52.067 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:52.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:52.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542182 2020-03-11T22:51:52.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-60129542182' 2020-03-11T22:51:52.192 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:52.192 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:52.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378660 2020-03-11T22:51:52.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-60129542182 3-81604378660' 2020-03-11T22:51:52.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:52.318 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:52.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215139 2020-03-11T22:51:52.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-60129542182 3-81604378660 4-103079215139' 2020-03-11T22:51:52.443 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:52.444 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836520 2020-03-11T22:51:52.444 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:52.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:52.445 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836520 2020-03-11T22:51:52.445 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:52.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836520 2020-03-11T22:51:52.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2020-03-11T22:51:52.446 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836520 2020-03-11T22:51:52.446 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:52.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836520 -lt 21474836520 2020-03-11T22:51:52.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:52.806 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672999 2020-03-11T22:51:52.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:52.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:52.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672999 2020-03-11T22:51:52.808 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:52.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672999 2020-03-11T22:51:52.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2020-03-11T22:51:52.809 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672999 2020-03-11T22:51:52.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:53.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672999 -lt 42949672999 2020-03-11T22:51:53.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:53.166 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542182 2020-03-11T22:51:53.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:53.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:53.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542182 2020-03-11T22:51:53.168 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:53.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542182 2020-03-11T22:51:53.169 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542182' 2020-03-11T22:51:53.169 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:53.169 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542182 2020-03-11T22:51:53.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542182 -lt 60129542182 2020-03-11T22:51:53.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:53.512 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378660 2020-03-11T22:51:53.512 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:53.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:53.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378660 2020-03-11T22:51:53.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378660 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378660' 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:53.514 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378660 2020-03-11T22:51:53.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378661 -lt 81604378660 2020-03-11T22:51:53.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:53.892 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215139 2020-03-11T22:51:53.892 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:53.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:53.893 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215139 2020-03-11T22:51:53.893 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:53.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215139 2020-03-11T22:51:53.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215139' 2020-03-11T22:51:53.894 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:53.894 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215139 2020-03-11T22:51:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215139 -lt 103079215139 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:51:54.240 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:51:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:54.333 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:51:54.333 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:51:54.333 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:54.333 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:54.333 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:51:54.334 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:51:54.335 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:51:54.336 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:54.337 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:54.338 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:51:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:54.656 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:54.657 INFO:tasks.workunit.client.0.smithi118.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:51:37.462573+0000 94'150 107:224 [1,4] 1 [1] 1 0'0 2020-03-11T22:50:02.558711+0000 0'0 2020-03-11T22:50:02.558711+0000 0 2020-03-11T22:51:54.657 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:51:44.792340+0000 94'150 107:236 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:49:48.661846+0000 0'0 2020-03-11T22:49:48.661846+0000 0 2020-03-11T22:51:54.657 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:51:50.899427+0000 94'150 108:236 [1,2] 1 [1] 1 0'0 2020-03-11T22:49:53.206251+0000 0'0 2020-03-11T22:49:53.206251+0000 0 2020-03-11T22:51:54.657 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:51:54.657 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:51:54.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:54.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:51:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:51:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:51:54.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:54.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:54.675 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:51:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:51:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:51:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:51:57.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:57.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:57.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:57.015 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:57.015 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:57.015 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:51:57.015 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:51:57.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:57.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:57.118 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:51:57.118 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:51:57.118 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:51:57.119 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 254, 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:51:57.120 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:57.121 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:57.122 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:51:57.123 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:51:57.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:57.125 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:57.468 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:57.468 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:57.469 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:57.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836523 2020-03-11T22:51:57.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523' 2020-03-11T22:51:57.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:57.618 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:57.745 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673002 2020-03-11T22:51:57.745 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673002' 2020-03-11T22:51:57.745 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:57.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:57.877 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542185 2020-03-11T22:51:57.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542185' 2020-03-11T22:51:57.878 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:57.878 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:57.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378664 2020-03-11T22:51:57.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542185 3-81604378664' 2020-03-11T22:51:57.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:57.998 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:58.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215142 2020-03-11T22:51:58.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542185 3-81604378664 4-103079215142' 2020-03-11T22:51:58.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:58.123 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836523 2020-03-11T22:51:58.123 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:58.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:58.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836523 2020-03-11T22:51:58.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:58.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836523 2020-03-11T22:51:58.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2020-03-11T22:51:58.126 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:58.126 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836523 2020-03-11T22:51:58.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836521 -lt 21474836523 2020-03-11T22:51:58.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:59.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:59.472 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:59.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836523 -lt 21474836523 2020-03-11T22:51:59.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:59.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673002 2020-03-11T22:51:59.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:59.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:59.829 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673002 2020-03-11T22:51:59.829 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:59.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673002 2020-03-11T22:51:59.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2020-03-11T22:51:59.830 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:59.831 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949673002 2020-03-11T22:52:00.186 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673002 -lt 42949673002 2020-03-11T22:52:00.187 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:00.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542185 2020-03-11T22:52:00.187 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:00.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542185 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542185 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542185' 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:00.191 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542185 2020-03-11T22:52:00.535 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542185 -lt 60129542185 2020-03-11T22:52:00.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:00.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378664 2020-03-11T22:52:00.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:00.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:00.537 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378664 2020-03-11T22:52:00.537 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:00.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378664 2020-03-11T22:52:00.538 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378664' 2020-03-11T22:52:00.538 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:00.539 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378664 2020-03-11T22:52:00.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378664 -lt 81604378664 2020-03-11T22:52:00.888 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:00.888 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215142 2020-03-11T22:52:00.888 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:00.888 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:00.889 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215142 2020-03-11T22:52:00.889 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:00.890 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215142 2020-03-11T22:52:00.890 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215142' 2020-03-11T22:52:00.890 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:00.890 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215142 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215143 -lt 103079215142 2020-03-11T22:52:01.240 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:52:01.567 INFO:tasks.workunit.client.0.smithi118.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:52:01.568 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:52:01.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:52:03.578 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:03.578 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:03.578 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:03.579 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:03.681 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:03.681 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:03.681 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:03.682 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:03.683 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:03.684 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:03.685 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:03.686 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:03.687 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:03.687 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:03.687 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:03.687 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:03.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:52:03.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:52:03.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:52:03.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:52:03.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:03.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:52:03.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:52:03.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:52:03.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:52:04.017 INFO:tasks.workunit.client.0.smithi118.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:52:04.018 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:52:04.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:52:09.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:09.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:09.027 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:09.028 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:09.028 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:09.028 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:52:09.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:52:09.029 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:09.029 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:09.123 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:09.123 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:09.123 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:09.123 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:09.124 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:09.125 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:09.126 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:09.127 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:09.128 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:09.129 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:09.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:52:09.131 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:09.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:09.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:09.474 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:09.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836528 2020-03-11T22:52:09.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528' 2020-03-11T22:52:09.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:09.600 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:09.725 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673006 2020-03-11T22:52:09.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673006' 2020-03-11T22:52:09.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:09.726 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:09.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542189 2020-03-11T22:52:09.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673006 2-60129542189' 2020-03-11T22:52:09.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:09.851 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:09.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378668 2020-03-11T22:52:09.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673006 2-60129542189 3-81604378668' 2020-03-11T22:52:09.976 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:09.976 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215147 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673006 2-60129542189 3-81604378668 4-103079215147' 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836528 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:10.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:10.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836528 2020-03-11T22:52:10.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:10.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836528 2020-03-11T22:52:10.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2020-03-11T22:52:10.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:10.105 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836528 2020-03-11T22:52:10.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836525 -lt 21474836528 2020-03-11T22:52:10.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:11.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:11.453 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:11.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836528 -lt 21474836528 2020-03-11T22:52:11.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:11.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673006 2020-03-11T22:52:11.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:11.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:11.808 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673006 2020-03-11T22:52:11.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:11.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673006 2020-03-11T22:52:11.810 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2020-03-11T22:52:11.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:11.810 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949673006 2020-03-11T22:52:12.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673007 -lt 42949673006 2020-03-11T22:52:12.154 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:12.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542189 2020-03-11T22:52:12.155 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:12.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:12.156 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542189 2020-03-11T22:52:12.156 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:12.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542189 2020-03-11T22:52:12.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542189' 2020-03-11T22:52:12.157 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:12.157 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542189 2020-03-11T22:52:12.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542189 -lt 60129542189 2020-03-11T22:52:12.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:12.506 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378668 2020-03-11T22:52:12.506 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:12.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:12.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378668 2020-03-11T22:52:12.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:12.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378668 2020-03-11T22:52:12.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378668' 2020-03-11T22:52:12.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:12.509 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378668 2020-03-11T22:52:12.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378669 -lt 81604378668 2020-03-11T22:52:12.852 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:12.852 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215147 2020-03-11T22:52:12.852 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:12.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:12.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215147 2020-03-11T22:52:12.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:12.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215147 2020-03-11T22:52:12.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215147' 2020-03-11T22:52:12.854 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:12.854 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215147 2020-03-11T22:52:13.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215147 -lt 103079215147 2020-03-11T22:52:13.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:52:13.564 INFO:tasks.workunit.client.0.smithi118.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:52:13.565 INFO:tasks.workunit.client.0.smithi118.stderr: 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:15.575 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:52:15.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:52:15.576 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:15.576 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:15.680 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:15.680 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:15.680 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:15.681 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:15.682 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:15.683 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:15.683 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:15.683 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:15.683 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:15.685 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:15.685 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:15.685 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:15.685 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:15.686 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:15.687 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:15.688 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:15.688 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:15.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:52:16.337 INFO:tasks.workunit.client.0.smithi118.stderr:noout is unset 2020-03-11T22:52:16.345 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:52:17.172 INFO:tasks.workunit.client.0.smithi118.stderr:nobackfill is unset 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:17.182 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:17.183 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:17.184 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:17.184 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:17.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:17.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:17.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:17.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:17.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:17.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:17.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:17.273 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:17.621 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:17.621 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr:4' 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:17.622 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:17.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836531 2020-03-11T22:52:17.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531' 2020-03-11T22:52:17.764 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:17.764 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:17.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673010 2020-03-11T22:52:17.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673010' 2020-03-11T22:52:17.891 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:17.892 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:18.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542193 2020-03-11T22:52:18.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542193' 2020-03-11T22:52:18.020 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:18.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:18.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378672 2020-03-11T22:52:18.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542193 3-81604378672' 2020-03-11T22:52:18.147 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:18.148 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:18.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215150 2020-03-11T22:52:18.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542193 3-81604378672 4-103079215150' 2020-03-11T22:52:18.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:18.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836531 2020-03-11T22:52:18.272 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:18.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:18.273 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836531 2020-03-11T22:52:18.273 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:18.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836531 2020-03-11T22:52:18.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2020-03-11T22:52:18.274 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:18.274 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836531 2020-03-11T22:52:18.718 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836531 -lt 21474836531 2020-03-11T22:52:18.718 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:18.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673010 2020-03-11T22:52:18.718 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:18.719 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:18.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673010 2020-03-11T22:52:18.719 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:18.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673010 2020-03-11T22:52:18.720 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2020-03-11T22:52:18.720 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:18.721 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949673010 2020-03-11T22:52:19.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673010 -lt 42949673010 2020-03-11T22:52:19.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:19.074 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542193 2020-03-11T22:52:19.075 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:19.075 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:19.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542193 2020-03-11T22:52:19.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:19.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542193 2020-03-11T22:52:19.077 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542193' 2020-03-11T22:52:19.077 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:19.077 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 60129542193 2020-03-11T22:52:19.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542193 -lt 60129542193 2020-03-11T22:52:19.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:19.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378672 2020-03-11T22:52:19.459 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:19.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:19.459 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378672 2020-03-11T22:52:19.460 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:19.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378672 2020-03-11T22:52:19.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378672' 2020-03-11T22:52:19.461 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 81604378672 2020-03-11T22:52:19.461 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:19.833 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378672 -lt 81604378672 2020-03-11T22:52:19.833 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:19.833 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215150 2020-03-11T22:52:19.833 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:19.834 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:19.834 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215150 2020-03-11T22:52:19.835 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:19.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215150 2020-03-11T22:52:19.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215150' 2020-03-11T22:52:19.836 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:19.836 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 103079215150 2020-03-11T22:52:20.266 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215150 -lt 103079215150 2020-03-11T22:52:20.266 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:20.267 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:20.267 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:20.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:20.766 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:21.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:21.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:21.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:21.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:21.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:21.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:52:21.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:21.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:52:21.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:21.553 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:21.553 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:21.641 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:21.641 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:21.641 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:21.642 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:21.643 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:21.644 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:21.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:21.646 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:21.647 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:21.647 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:21.647 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:21.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:21.750 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:21.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:21.751 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:22.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:22.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:22.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:22.085 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:22.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:22.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:22.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:22.650 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:23.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=8583452 2020-03-11T22:52:23.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 8583452 '!=' null 2020-03-11T22:52:23.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:23.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.198 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:23.199 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.286 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:23.286 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:23.286 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:23.286 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:23.286 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:23.287 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:23.288 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:23.289 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:23.290 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:23.291 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:23.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:23.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:23.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:23.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:23.731 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:23.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:23.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:24.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:24.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:24.182 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:24.939 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=6552431 2020-03-11T22:52:24.939 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 6552431 '!=' null 2020-03-11T22:52:24.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:24.940 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:24.940 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:24.940 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:25.027 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:25.027 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:25.027 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:25.027 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:25.028 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:25.029 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:25.030 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:25.031 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:25.032 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:25.033 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:25.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:25.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:25.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:25.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:25.136 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:25.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:25.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:25.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:25.137 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:25.472 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:25.472 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:25.472 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:25.472 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:25.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:25.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:25.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:25.923 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:26.383 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=6552431 2020-03-11T22:52:26.383 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 6552431 '!=' null 2020-03-11T22:52:26.383 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:26.383 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:26.384 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.473 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:26.474 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:26.475 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:26.476 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:26.477 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:26.478 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:26.479 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:26.479 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:26.479 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:26.479 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:26.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:26.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:26.582 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:26.582 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:26.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:26.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:26.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:26.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:26.583 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:26.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:26.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:26.929 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:26.930 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:27.373 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:27.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:27.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:27.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:27.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:27.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:27.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:27.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:27.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:27.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17432030 2020-03-11T22:52:27.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17432030 '!=' null 2020-03-11T22:52:27.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:27.828 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:27.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:27.829 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:27.915 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:27.915 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:27.915 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:27.916 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:27.917 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:27.918 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:27.919 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:27.920 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:27.921 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:27.921 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:27.921 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:27.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:28.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:28.025 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:28.025 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:28.362 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:28.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:28.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:28.363 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:28.811 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:28.812 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:28.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:29.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14881657 2020-03-11T22:52:29.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14881657 '!=' null 2020-03-11T22:52:29.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:29.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:29.371 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:29.372 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:29.458 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:29.458 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:29.458 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:29.459 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:29.459 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:29.459 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:29.459 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:29.459 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:29.460 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:29.460 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:29.460 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:29.460 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:29.460 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:29.461 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:29.461 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:29.461 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:29.461 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:29.462 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:29.462 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:29.462 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:29.462 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:29.462 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:29.463 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:29.463 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:29.463 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:29.464 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:29.464 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:29.464 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:29.464 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:29.464 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:29.465 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:29.465 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:29.465 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:29.465 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:29.465 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:29.466 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:29.466 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:29.466 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:29.466 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:29.466 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:29.467 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:29.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:29.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:29.567 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:29.567 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:29.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:29.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:29.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:29.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:29.569 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:29.910 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:29.910 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:29.910 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:29.911 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:30.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:30.359 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:30.359 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:30.359 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:30.810 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25335823 2020-03-11T22:52:30.811 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25335823 '!=' null 2020-03-11T22:52:30.811 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:30.811 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:30.811 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:30.811 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:30.897 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:30.898 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:30.898 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:30.898 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:30.898 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:30.898 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:30.899 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:30.899 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:30.899 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:30.899 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:30.899 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:30.900 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:30.900 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:30.900 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:30.900 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:30.900 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:30.901 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:30.901 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:30.901 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:30.901 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:30.901 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:30.902 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:30.902 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:30.902 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:30.902 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:30.902 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:30.903 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:30.903 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:30.903 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:30.903 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:30.903 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:30.904 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:30.904 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:30.904 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:30.904 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:30.904 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:30.905 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:30.905 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:30.905 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:30.905 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:30.905 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:30.906 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:31.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:31.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:31.007 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:31.008 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:31.008 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:31.008 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:31.008 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:31.008 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:31.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:31.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:31.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:31.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:31.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:31.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:31.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:31.786 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:31.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:31.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:31.787 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:31.787 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:31.787 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:32.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25335823 2020-03-11T22:52:32.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25335823 '!=' null 2020-03-11T22:52:32.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:32.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:32.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:32.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:32.325 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:32.326 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:32.327 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:32.328 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:32.329 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:32.330 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:32.330 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:32.330 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:32.330 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:32.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:32.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:32.432 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:32.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:32.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:32.433 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:32.434 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:32.434 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:32.434 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:32.769 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:32.769 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:32.770 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:32.770 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:33.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:33.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:33.223 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:33.223 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:33.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967548 2020-03-11T22:52:33.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967548 '!=' null 2020-03-11T22:52:33.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:33.756 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:33.757 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:33.757 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:33.844 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:33.844 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:33.844 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:33.844 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:33.845 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:33.846 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:33.847 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:33.847 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:33.847 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:33.847 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:33.847 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:33.848 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:33.849 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:33.849 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:33.849 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:33.849 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:33.849 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:33.850 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:33.850 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:33.850 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:33.850 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:33.850 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:33.851 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:33.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:33.953 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:33.953 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:33.953 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:33.953 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:33.954 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:33.954 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:33.954 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:34.295 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:34.295 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:34.295 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:34.295 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:34.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:34.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:34.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:34.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:34.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:34.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:34.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:34.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:34.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:35.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967577 2020-03-11T22:52:35.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967577 '!=' null 2020-03-11T22:52:35.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:35.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:35.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:35.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:35.286 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:35.286 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:35.286 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:35.286 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:35.287 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:35.288 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:35.289 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:35.290 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:35.291 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:35.292 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:35.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:35.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:35.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:35.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:35.731 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:35.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:35.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:35.732 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:36.178 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:36.179 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:36.180 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:36.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451456 2020-03-11T22:52:36.715 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451456 '!=' null 2020-03-11T22:52:36.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:36.716 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:36.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:36.716 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:36.802 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:36.803 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:36.804 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:36.805 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:36.806 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:36.807 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:36.808 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:36.808 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:36.808 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:36.808 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:36.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:36.911 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:36.911 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:36.912 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:37.250 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:37.251 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:37.251 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:37.251 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:37.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:37.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:37.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:37.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:37.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:37.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:37.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:37.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:37.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:38.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451456 2020-03-11T22:52:38.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451456 '!=' null 2020-03-11T22:52:38.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:38.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:38.201 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:38.201 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:38.289 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:38.290 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:38.291 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:38.292 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:38.292 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:38.292 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 151, 2020-03-11T22:52:38.292 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:38.292 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:38.293 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "7.0", 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 254, 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:38.294 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:38.295 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:38.296 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:38.296 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:38.296 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:38.296 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:38.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:38.398 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:38.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:38.398 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:38.399 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:38.399 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:38.399 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:38.399 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:38.796 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:38.796 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:38.796 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:38.797 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:39.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:39.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:39.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:39.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841485 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841485 '!=' null 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:39.907 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:39.993 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:39.994 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:39.995 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:39.996 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:39.997 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:39.998 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:40.000 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:40.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:40.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:40.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:40.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:40.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:40.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:40.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:40.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:40.438 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:40.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:40.438 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:40.439 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:41.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:41.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:41.052 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:41.053 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:41.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141193 2020-03-11T22:52:41.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141193 '!=' null 2020-03-11T22:52:41.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:41.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:41.510 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:41.510 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:41.597 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:41.597 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:41.598 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:41.599 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:41.600 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:41.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:41.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:41.706 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:41.706 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:41.706 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:41.707 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:41.707 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:41.707 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:41.707 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:42.043 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:42.044 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:42.044 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:42.044 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:42.494 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:42.495 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:42.582 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:42.583 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:42.584 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:42.585 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:42.586 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:42.590 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:42.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:42.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:42.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:42.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:42.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:42.692 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:42.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:42.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:43.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:43.045 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:43.046 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:43.046 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:43.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:43.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:43.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:43.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:43.533 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:43.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:43.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:43.534 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:43.534 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:44.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141378 2020-03-11T22:52:44.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141378 '!=' null 2020-03-11T22:52:44.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:44.019 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:44.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:44.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:44.107 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:44.107 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:44.107 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:44.107 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:44.108 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:44.109 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:44.110 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:44.111 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:44.111 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:44.111 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:44.111 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:44.111 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:44.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:44.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:44.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:44.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:44.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:44.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:44.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:44.218 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:44.218 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:44.560 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:44.560 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:44.561 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:44.561 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:45.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:45.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:45.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:45.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:45.013 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:45.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:45.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:45.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:45.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:45.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415289 2020-03-11T22:52:45.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415289 '!=' null 2020-03-11T22:52:45.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:45.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:45.546 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:45.546 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:45.631 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:45.631 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:45.631 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:45.632 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:45.633 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:45.634 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:45.635 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:45.636 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:45.636 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:45.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:45.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:45.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:45.742 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:46.075 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:46.075 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:46.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:46.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:46.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:46.531 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:46.532 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:46.533 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:46.998 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141188 2020-03-11T22:52:46.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141188 '!=' null 2020-03-11T22:52:46.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:46.999 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:46.999 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:46.999 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:47.087 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:47.087 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:47.087 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:47.087 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:47.088 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:47.089 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:47.090 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:47.091 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:47.091 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:47.091 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:47.091 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:47.091 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:47.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:47.195 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:47.195 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:47.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:47.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:47.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:47.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:47.196 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:47.197 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:47.535 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:47.535 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:47.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:47.536 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:47.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:47.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:47.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:47.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:47.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:47.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:47.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:47.988 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:47.988 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:48.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141188 2020-03-11T22:52:48.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141188 '!=' null 2020-03-11T22:52:48.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:48.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:48.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:48.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:48.648 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:48.649 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:48.650 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:48.651 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:48.652 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:48.653 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:48.653 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:48.653 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:48.656 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:48.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:48.757 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:48.758 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:48.758 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:48.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:48.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:48.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:48.759 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:49.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:49.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:49.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:49.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:49.545 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:49.547 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:49.547 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:50.111 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24894767 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24894767 '!=' null 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:50.198 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:50.199 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:50.200 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:50.201 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:50.202 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:50.203 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:50.203 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:50.203 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:50.205 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:50.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:50.307 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:50.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:50.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:50.308 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:50.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:50.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:50.309 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:50.648 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:50.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:50.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:50.649 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:51.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:51.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:51.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:51.100 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:51.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:51.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:51.101 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:51.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:51.101 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:51.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32515952 2020-03-11T22:52:51.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32515952 '!=' null 2020-03-11T22:52:51.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:51.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:51.548 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:51.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:51.635 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:51.636 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:51.637 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:51.638 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:51.639 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:51.640 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:51.640 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:51.640 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:51.640 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:51.642 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:51.744 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:51.744 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:51.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:51.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:51.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:51.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:51.745 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:51.746 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:52.081 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:52.081 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:52.082 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:52.082 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:52.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:52.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:52.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:52.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:53.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967823 2020-03-11T22:52:53.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967823 '!=' null 2020-03-11T22:52:53.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:53.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:53.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:53.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:53.208 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:53.208 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:53.208 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:53.208 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:53.208 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:53.209 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:53.210 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:53.211 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:53.212 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:53.212 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:53.212 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:53.212 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:53.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:53.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:53.316 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:53.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:53.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:53.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:53.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:53.317 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:53.318 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:53.652 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:53.653 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:53.653 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:53.653 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:54.102 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:54.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:54.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:54.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967823 2020-03-11T22:52:54.555 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967823 '!=' null 2020-03-11T22:52:54.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:54.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:54.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:54.556 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:54.643 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:54.643 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:54.644 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:54.645 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:54.646 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:54.647 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:54.752 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:54.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:54.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:54.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:54.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:54.753 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:54.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:54.754 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:55.088 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:55.089 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:55.089 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:55.089 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:55.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:55.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:55.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:55.586 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:56.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967806 2020-03-11T22:52:56.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967806 '!=' null 2020-03-11T22:52:56.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:56.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:56.035 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:56.035 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:56.122 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:56.124 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:56.125 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:56.126 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:56.126 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:56.126 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:56.126 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:56.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:56.231 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:56.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:56.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:56.567 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:56.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:56.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:56.568 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:57.023 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:57.023 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:57.024 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:57.543 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451641 2020-03-11T22:52:57.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451641 '!=' null 2020-03-11T22:52:57.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:57.544 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:57.544 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:57.544 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:57.631 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:57.632 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:57.633 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:57.634 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:57.635 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:57.635 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:57.635 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:57.635 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:57.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:57.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:57.739 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:57.740 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:57.740 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:57.740 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:57.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:57.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:57.741 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:58.076 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:58.076 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:58.077 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:58.077 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:58.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:58.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:58.527 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:58.528 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:59.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967697 2020-03-11T22:52:59.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967697 '!=' null 2020-03-11T22:52:59.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:59.039 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:59.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:59.040 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:59.128 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:52:59.128 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:52:59.128 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:59.128 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:59.128 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [ 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: "priority": 110, 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: "items": [ 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:52:59.129 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:52:59.130 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "3.0", 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 151, 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:52:59.131 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:52:59.132 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:52:59.132 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:52:59.132 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:52:59.132 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:52:59.132 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:52:59.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:59.235 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:59.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:59.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:59.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:59.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:59.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:59.237 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:59.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:59.574 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:59.574 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:59.574 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:59.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:00.025 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:00.026 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:00.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:00.026 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:00.760 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967697 2020-03-11T22:53:00.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967697 '!=' null 2020-03-11T22:53:00.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:00.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:00.761 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:00.761 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:00.848 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:00.849 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:00.849 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:00.849 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:00.849 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:00.849 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:00.850 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:00.851 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:00.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:00.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:00.958 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:00.959 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:01.291 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:01.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:01.292 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:01.292 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:01.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:53:01.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:53:01.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:01.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:01.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:01.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:01.777 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:01.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:01.778 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:02.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25161295 2020-03-11T22:53:02.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25161295 '!=' null 2020-03-11T22:53:02.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:02.315 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:02.315 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:02.316 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:02.403 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:02.404 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:02.405 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:02.410 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:02.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:02.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:02.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:02.850 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:02.851 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:02.851 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:02.851 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:03.376 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:53:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:53:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:03.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:03.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:03.378 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:03.378 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:03.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451130 2020-03-11T22:53:03.830 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451130 '!=' null 2020-03-11T22:53:03.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:03.831 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:03.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:03.831 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:03.919 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:03.919 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:03.920 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:03.920 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:03.920 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:03.920 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:03.921 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:03.921 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:03.921 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:03.921 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:03.922 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:03.922 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:03.922 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:03.922 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:03.922 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:03.923 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:03.923 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:03.923 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:03.923 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:03.924 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:03.926 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:04.028 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:04.029 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:04.029 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:04.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:04.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:04.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:04.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:04.862 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:53:04.862 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:04.863 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:04.864 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:05.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725701 2020-03-11T22:53:05.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725701 '!=' null 2020-03-11T22:53:05.318 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:05.319 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:05.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:05.319 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:05.407 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:05.407 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:05.407 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:05.407 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:05.408 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:05.409 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:05.410 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:05.410 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:05.410 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:05.414 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:05.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:05.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:05.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:05.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:05.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:05.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:05.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:05.518 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:05.853 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:05.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:05.853 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:05.854 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:06.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:53:06.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:53:06.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:06.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:06.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:06.309 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:06.310 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:06.310 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:06.310 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:06.932 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141228 2020-03-11T22:53:06.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141228 '!=' null 2020-03-11T22:53:06.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:06.933 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:06.933 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:06.934 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:07.021 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:07.021 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:07.022 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:07.022 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:07.022 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:07.022 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:07.022 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:07.023 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:07.023 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:07.023 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:07.023 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:07.024 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:07.024 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:07.024 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:07.024 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:07.024 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:07.025 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:07.025 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:07.025 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:07.025 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:07.028 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:07.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:07.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:07.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:07.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:07.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:07.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:07.132 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:07.132 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:07.467 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:07.468 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:07.468 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:07.468 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:07.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:07.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:53:07.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:53:07.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:53:07.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:07.920 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:07.920 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:08.005 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:08.006 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:08.007 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:08.008 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:08.009 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:08.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:08.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:08.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:08.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:08.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:08.457 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:08.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:08.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:09.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:09.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:09.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:09.036 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:09.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:09.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:09.037 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:09.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:09.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:09.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14102880 2020-03-11T22:53:09.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14102880 '!=' null 2020-03-11T22:53:09.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:09.575 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:09.575 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:09.576 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:09.662 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:09.663 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:09.664 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:09.668 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:09.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:09.770 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:09.770 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:09.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:09.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:09.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:09.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:09.771 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:10.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:10.107 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:10.108 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:10.108 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:10.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:10.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:10.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:10.609 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:11.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23064695 2020-03-11T22:53:11.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23064695 '!=' null 2020-03-11T22:53:11.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:11.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:11.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:11.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:11.232 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:11.232 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:11.232 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:11.232 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:11.232 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:11.233 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:11.234 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:11.235 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:11.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:11.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:11.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:11.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:11.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:11.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:11.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:11.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:11.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:11.679 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:11.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:11.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:11.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:12.128 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:12.128 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:12.129 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:12.580 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23064695 2020-03-11T22:53:12.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23064695 '!=' null 2020-03-11T22:53:12.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:12.581 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:12.581 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:12.581 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:12.667 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:12.667 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:12.667 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:12.668 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:12.669 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:12.670 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:12.670 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:12.670 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:12.670 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:12.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:12.776 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:12.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:12.777 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:13.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:13.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:13.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:13.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:13.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:13.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:13.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:13.563 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:14.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19628047 2020-03-11T22:53:14.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19628047 '!=' null 2020-03-11T22:53:14.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:14.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:14.168 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:14.169 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:14.257 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:14.258 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:14.259 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:14.260 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:14.264 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:14.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:14.367 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:14.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:14.367 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:14.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:14.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:14.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:14.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:14.698 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:14.698 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:14.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:14.699 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:15.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:15.151 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:15.152 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:15.609 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220832 2020-03-11T22:53:15.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220832 '!=' null 2020-03-11T22:53:15.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:15.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:15.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:15.610 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:15.698 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:15.698 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:15.698 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:15.698 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:15.698 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:15.699 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:15.700 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:15.701 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:15.705 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:15.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:15.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:15.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:15.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:15.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:15.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:15.807 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:15.808 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:16.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:16.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:16.146 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:16.146 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:16.600 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:16.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:16.601 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:16.601 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831228 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831228 '!=' null 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:17.144 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:17.232 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:17.232 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:17.232 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:17.232 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:17.233 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:17.234 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:17.235 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:17.235 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:17.235 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:17.235 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:17.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:17.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:17.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:17.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:17.342 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:17.677 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:17.677 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:17.677 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:17.678 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:18.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:18.133 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:18.134 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:18.135 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:18.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831228 2020-03-11T22:53:18.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831228 '!=' null 2020-03-11T22:53:18.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:18.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:18.604 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:18.604 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:18.692 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:18.693 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:18.694 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:18.695 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:18.695 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:18.695 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:18.695 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:18.695 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:18.699 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:18.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:18.801 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:18.802 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:19.138 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:19.139 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:19.139 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:19.139 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:19.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:19.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:19.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:19.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:19.589 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:19.590 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:19.590 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:19.590 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:19.590 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:20.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220774 2020-03-11T22:53:20.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220774 '!=' null 2020-03-11T22:53:20.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:20.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:20.035 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:20.035 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:20.123 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:20.123 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:20.123 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:20.124 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:20.125 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:20.126 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:20.126 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:20.126 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:20.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:20.232 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:20.233 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:20.233 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:20.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:20.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:20.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:20.571 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:21.015 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:21.016 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:21.016 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:21.016 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704974 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704974 '!=' null 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:21.471 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:21.557 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [ 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:21.558 INFO:tasks.workunit.client.0.smithi118.stdout: "item": "1.0", 2020-03-11T22:53:21.559 INFO:tasks.workunit.client.0.smithi118.stdout: "prio": 110, 2020-03-11T22:53:21.559 INFO:tasks.workunit.client.0.smithi118.stdout: "can_preempt": true 2020-03-11T22:53:21.559 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:21.561 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:21.562 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:21.562 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:21.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:21.665 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:21.666 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:21.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:21.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:21.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:21.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:21.667 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:21.668 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:22.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:22.007 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:22.007 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:22.007 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:22.454 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:22.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:22.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:22.455 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24113507 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24113507 '!=' null 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:22.950 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:23.035 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:23.036 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:23.037 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:23.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:23.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:23.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:23.144 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:23.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:23.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:23.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:23.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:23.145 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:23.476 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:23.476 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:23.477 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:23.477 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:23.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:23.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:23.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:23.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:24.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24113507 2020-03-11T22:53:24.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24113507 '!=' null 2020-03-11T22:53:24.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:24.368 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:24.368 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:24.369 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:24.455 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:24.456 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:24.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:24.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:24.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:24.564 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:24.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:24.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:24.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:24.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:24.565 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:24.895 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:24.895 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:24.896 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:24.896 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:25.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:25.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:25.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:25.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:25.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:25.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:25.341 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:25.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:25.341 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:25.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15726098 2020-03-11T22:53:25.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15726098 '!=' null 2020-03-11T22:53:25.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:25.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:25.870 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:25.871 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:25.956 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:25.956 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:25.956 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:25.956 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:25.956 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:25.957 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:25.958 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:25.958 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:25.962 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:26.064 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:26.064 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:26.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:26.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:26.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:26.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:26.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:26.067 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:26.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:26.394 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:26.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:26.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:26.841 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:26.841 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:26.841 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:26.842 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:27.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22278641 2020-03-11T22:53:27.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22278641 '!=' null 2020-03-11T22:53:27.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:27.283 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:27.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:27.284 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:27.370 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "local_reservations": { 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:27.371 INFO:tasks.workunit.client.0.smithi118.stdout: "remote_reservations": { 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout: "max_allowed": 1, 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout: "min_priority": 0, 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout: "queues": [], 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout: "in_progress": [] 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:27.372 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:27.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:27.479 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:27.480 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:27.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:53:27.812 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:27.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:27.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:28.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:53:28.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:28.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:28.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:28.627 INFO:tasks.workunit.client.0.smithi118.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:28.627 INFO:tasks.workunit.client.0.smithi118.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:39.317573+0000 94'150 117:348 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:50:02.558711+0000 0'0 2020-03-11T22:50:02.558711+0000 0 2020-03-11T22:53:28.627 INFO:tasks.workunit.client.0.smithi118.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:53:22.703711+0000 94'150 118:357 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:48.661846+0000 0'0 2020-03-11T22:49:48.661846+0000 0 2020-03-11T22:53:28.627 INFO:tasks.workunit.client.0.smithi118.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:53:00.763772+0000 94'150 117:362 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:53.206251+0000 0'0 2020-03-11T22:49:53.206251+0000 0 2020-03-11T22:53:28.627 INFO:tasks.workunit.client.0.smithi118.stdout: 2020-03-11T22:53:28.628 INFO:tasks.workunit.client.0.smithi118.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:28.628 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:53:28.637 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55883/ceph-osd.1.asok 2020-03-11T22:53:28.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:28.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55883/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout: "pgs": [ 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout: "pg": "7.0", 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.726 INFO:tasks.workunit.client.0.smithi118.stdout: "history": [ 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "61", 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.701415+0000", 2020-03-11T22:53:28.727 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:02.701438+0000" 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.701453+0000", 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:02.802032+0000" 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.728 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.802033+0000", 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:02.802135+0000" 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.729 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.802136+0000", 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:02.802147+0000" 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.802148+0000", 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:03.027988+0000" 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.730 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.701446+0000", 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:03.027998+0000" 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.731 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:03.028250+0000", 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:03.315039+0000" 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:03.315039+0000", 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:03.315087+0000" 2020-03-11T22:53:28.732 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:03.315087+0000", 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.649925+0000" 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.733 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:03.027998+0000", 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.649945+0000" 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.701439+0000", 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.649962+0000" 2020-03-11T22:53:28.734 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:02.701408+0000", 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.649968+0000" 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.735 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "61", 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.649968+0000", 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.650158+0000" 2020-03-11T22:53:28.736 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "92", 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.737 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.738 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.738 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.650170+0000", 2020-03-11T22:53:28.738 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.650183+0000" 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.650191+0000", 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.690553+0000" 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.739 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.690553+0000", 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.690634+0000" 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.740 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.690635+0000", 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:04.690649+0000" 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.690649+0000", 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:05.010614+0000" 2020-03-11T22:53:28.741 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.650187+0000", 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:05.010631+0000" 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.742 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:05.010837+0000", 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:05.408402+0000" 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.743 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:05.408402+0000", 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:05.408450+0000" 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:05.408450+0000", 2020-03-11T22:53:28.744 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955368+0000" 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:05.010632+0000", 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955385+0000" 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.745 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.650183+0000", 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955400+0000" 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.746 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:04.650158+0000", 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955404+0000" 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "92", 2020-03-11T22:53:28.747 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955404+0000", 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955572+0000" 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.748 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "99", 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955577+0000", 2020-03-11T22:53:28.749 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.955587+0000" 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955594+0000", 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.035559+0000" 2020-03-11T22:53:28.750 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.035560+0000", 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.035616+0000" 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.751 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.035617+0000", 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.035627+0000" 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.752 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.035627+0000", 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.202123+0000" 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955591+0000", 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.202131+0000" 2020-03-11T22:53:28.753 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.202202+0000", 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.220374+0000" 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.754 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.220374+0000", 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:30.220414+0000" 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.220414+0000", 2020-03-11T22:53:28.755 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070175+0000" 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:30.202132+0000", 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070209+0000" 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.756 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955588+0000", 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070257+0000" 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.955572+0000", 2020-03-11T22:53:28.757 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070266+0000" 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "100", 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.758 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070266+0000", 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070762+0000" 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.759 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "101", 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070770+0000", 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.070795+0000" 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.760 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070810+0000", 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.102014+0000" 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.761 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.102014+0000", 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.102089+0000" 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070803+0000", 2020-03-11T22:53:28.762 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.102105+0000" 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.102106+0000", 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.256653+0000" 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.763 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070796+0000", 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.256681+0000" 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.764 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.070762+0000", 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.256690+0000" 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.765 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "101", 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.256690+0000", 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.257050+0000" 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.766 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "113", 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.767 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.257059+0000", 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.257084+0000" 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.257100+0000", 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.382195+0000" 2020-03-11T22:53:28.768 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.382195+0000", 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.382325+0000" 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.769 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.382326+0000", 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.382356+0000" 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.382357+0000", 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.443824+0000" 2020-03-11T22:53:28.770 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.257092+0000", 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.443834+0000" 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.771 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.444065+0000", 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.723699+0000" 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.723699+0000", 2020-03-11T22:53:28.772 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.723845+0000" 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.723845+0000", 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:31.724267+0000" 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.773 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.724268+0000", 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:01.471534+0000" 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:01.471534+0000", 2020-03-11T22:53:28.774 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:01.471699+0000" 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:01.471699+0000", 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:13.468871+0000" 2020-03-11T22:53:28.775 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:13.468871+0000", 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:13.469257+0000" 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.776 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:13.469257+0000", 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.655564+0000" 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:38.655565+0000", 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.655737+0000" 2020-03-11T22:53:28.777 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:38.655738+0000", 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.775539+0000" 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.778 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.443835+0000", 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.775569+0000" 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.779 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.257084+0000", 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.775589+0000" 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:31.257050+0000", 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.775595+0000" 2020-03-11T22:53:28.780 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "113", 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.781 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:38.775600+0000", 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.775878+0000" 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.782 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: "pg": "3.0", 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: "history": [ 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "38", 2020-03-11T22:53:28.783 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.700357+0000", 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.700377+0000" 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.784 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.700388+0000", 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.806103+0000" 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.806104+0000", 2020-03-11T22:53:28.785 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.806169+0000" 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.806170+0000", 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.806184+0000" 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.786 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.806184+0000", 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.894937+0000" 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.787 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.700383+0000", 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:53.894955+0000" 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.895216+0000", 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:54.514336+0000" 2020-03-11T22:53:28.788 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:54.514337+0000", 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:54.514387+0000" 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.789 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:54.514387+0000", 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.086848+0000" 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.894955+0000", 2020-03-11T22:53:28.790 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.086881+0000" 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.700378+0000", 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.086907+0000" 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.791 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:53.700352+0000", 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.086914+0000" 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.792 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "38", 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.086914+0000", 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.087073+0000" 2020-03-11T22:53:28.793 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "89", 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.794 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087077+0000", 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.087087+0000" 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087093+0000", 2020-03-11T22:53:28.795 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.087517+0000" 2020-03-11T22:53:28.796 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.796 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.796 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.796 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087518+0000", 2020-03-11T22:53:28.796 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.087572+0000" 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087573+0000", 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.087585+0000" 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.797 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087585+0000", 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.388514+0000" 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.798 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087090+0000", 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.388522+0000" 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.388653+0000", 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.514858+0000" 2020-03-11T22:53:28.799 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.514858+0000", 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:55.514904+0000" 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.800 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.514904+0000", 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438280+0000" 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.388522+0000", 2020-03-11T22:53:28.801 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438299+0000" 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087087+0000", 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438314+0000" 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.802 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:55.087073+0000", 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438318+0000" 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.803 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "89", 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438321+0000", 2020-03-11T22:53:28.804 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438493+0000" 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "106", 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.805 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438497+0000", 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.438508+0000" 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.806 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438515+0000", 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.497222+0000" 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.497222+0000", 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.497273+0000" 2020-03-11T22:53:28.807 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.497273+0000", 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.497283+0000" 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.808 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.809 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.809 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.497284+0000", 2020-03-11T22:53:28.809 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.837771+0000" 2020-03-11T22:53:28.809 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.810 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.810 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438512+0000", 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.837783+0000" 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.837859+0000", 2020-03-11T22:53:28.811 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.859259+0000" 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.859260+0000", 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:50:29.859338+0000" 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.812 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.859338+0000", 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316285+0000" 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.813 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.837784+0000", 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316331+0000" 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438509+0000", 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316384+0000" 2020-03-11T22:53:28.814 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:50:29.438494+0000", 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316391+0000" 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.815 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "106", 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316391+0000", 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316677+0000" 2020-03-11T22:53:28.816 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "107", 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.817 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316683+0000", 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.316697+0000" 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316706+0000", 2020-03-11T22:53:28.818 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.317106+0000" 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.317106+0000", 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.317154+0000" 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.819 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316702+0000", 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.317162+0000" 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.820 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.317163+0000", 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.482453+0000" 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316698+0000", 2020-03-11T22:53:28.821 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.482481+0000" 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.316677+0000", 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.482490+0000" 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.822 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "107", 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.823 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482490+0000", 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.482889+0000" 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "115", 2020-03-11T22:53:28.824 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482897+0000", 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.482923+0000" 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482955+0000", 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.628206+0000" 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.628207+0000", 2020-03-11T22:53:28.827 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.628282+0000" 2020-03-11T22:53:28.827 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.827 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.827 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.827 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.628283+0000", 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.628300+0000" 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.628300+0000", 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.843238+0000" 2020-03-11T22:53:28.828 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482947+0000", 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:49.843248+0000" 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.829 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.843482+0000", 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:50.899382+0000" 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:28.830 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:50.899382+0000", 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:01.471615+0000" 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:01.471616+0000", 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:01.471959+0000" 2020-03-11T22:53:28.831 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:01.471959+0000", 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:13.468763+0000" 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.832 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:13.468763+0000", 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:13.468987+0000" 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:13.468988+0000", 2020-03-11T22:53:28.833 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.655920+0000" 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:38.655920+0000", 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:52:38.656396+0000" 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.834 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:52:38.656396+0000", 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.136766+0000" 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.835 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:00.136766+0000", 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.136915+0000" 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:00.136915+0000", 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.336787+0000" 2020-03-11T22:53:28.836 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.843248+0000", 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.336811+0000" 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.837 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.838 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482923+0000", 2020-03-11T22:53:28.838 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.336830+0000" 2020-03-11T22:53:28.838 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.838 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.838 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:49.482889+0000", 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.336836+0000" 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.839 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "115", 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:00.336837+0000", 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.337099+0000" 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.840 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: "pg": "1.0", 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: "history": [ 2020-03-11T22:53:28.841 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "25", 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Initial", 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746440+0000", 2020-03-11T22:53:28.842 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.746534+0000" 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "25", 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.843 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746535+0000", 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.746569+0000" 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.844 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "28", 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746575+0000", 2020-03-11T22:53:28.845 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.746586+0000" 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746592+0000", 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.820092+0000" 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.846 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.820092+0000", 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.820190+0000" 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.820191+0000", 2020-03-11T22:53:28.847 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:48.820211+0000" 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.820212+0000", 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:49.080499+0000" 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.848 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746589+0000", 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:49.080507+0000" 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.849 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:49.080737+0000", 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:49.918791+0000" 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:49.918791+0000", 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:49.918876+0000" 2020-03-11T22:53:28.850 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:49.918877+0000", 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.553739+0000" 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.851 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:49.080508+0000", 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.553771+0000" 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.852 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746586+0000", 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.553796+0000" 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:48.746569+0000", 2020-03-11T22:53:28.853 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.553804+0000" 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "29", 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.854 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.553805+0000", 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.554233+0000" 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.855 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "103", 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.554241+0000", 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.554267+0000" 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.856 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.554282+0000", 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.788025+0000" 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.788025+0000", 2020-03-11T22:53:28.857 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.788092+0000" 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.788092+0000", 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.788105+0000" 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.858 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.788105+0000", 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.863208+0000" 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.859 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.554274+0000", 2020-03-11T22:53:28.860 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.863218+0000" 2020-03-11T22:53:28.860 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.860 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.860 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.863370+0000", 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.872064+0000" 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.872064+0000", 2020-03-11T22:53:28.861 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:49:50.872114+0000" 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.872115+0000", 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558472+0000" 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.862 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.863219+0000", 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558501+0000" 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.863 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.554267+0000", 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558552+0000" 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:49:50.554234+0000", 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558559+0000" 2020-03-11T22:53:28.864 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "103", 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.865 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558560+0000", 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558905+0000" 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.866 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "104", 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558911+0000", 2020-03-11T22:53:28.867 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.558924+0000" 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558933+0000", 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.559657+0000" 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.868 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.559658+0000", 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.559736+0000" 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558928+0000", 2020-03-11T22:53:28.869 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.559746+0000" 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.559746+0000", 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.816366+0000" 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.870 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558924+0000", 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.816385+0000" 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.871 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.558905+0000", 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.816395+0000" 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "104", 2020-03-11T22:53:28.872 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816395+0000", 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.816652+0000" 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.873 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "117", 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Start", 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816657+0000", 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.816674+0000" 2020-03-11T22:53:28.874 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816681+0000", 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.958686+0000" 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.875 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.958687+0000", 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.958781+0000" 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.958782+0000", 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:43.958808+0000" 2020-03-11T22:53:28.876 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.958808+0000", 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:44.075722+0000" 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.877 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816678+0000", 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:44.075736+0000" 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:44.076084+0000", 2020-03-11T22:53:28.878 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:51:44.792303+0000" 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:44.792304+0000", 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.137082+0000" 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.879 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:00.137082+0000", 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:00.137480+0000" 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:00.137481+0000", 2020-03-11T22:53:28.880 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.164133+0000" 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:22.164134+0000", 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.164327+0000" 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.881 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:22.164327+0000", 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.335950+0000" 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:28.882 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:44.075737+0000", 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.335983+0000" 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started/Primary", 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816674+0000", 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.336013+0000" 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.883 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Started", 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:51:43.816653+0000", 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.336022+0000" 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:53:28.884 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: "epoch": "117", 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: "states": [ 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: "state": "Reset", 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: "enter": "2020-03-11T22:53:22.336023+0000", 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: "exit": "2020-03-11T22:53:22.336431+0000" 2020-03-11T22:53:28.885 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:53:28.886 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:53:28.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:53:28.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:53:28.887 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:53:28.888 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:53:28.888 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:53:28.888 INFO:tasks.workunit.client.0.smithi118.stdout:TEST PASSED 2020-03-11T22:53:29.208 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test1' does not exist 2020-03-11T22:53:29.216 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:53:29.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:53:29.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:53:29.739 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test3' does not exist 2020-03-11T22:53:29.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:53:29.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:53:29.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:53:30.213 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test7' does not exist 2020-03-11T22:53:30.221 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:30.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:30.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:53:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:53:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:53:30.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:30.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:53:30.334 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:30.334 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:30.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:30.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:30.335 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:30.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:30.338 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:30.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:30.339 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:30.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:30.340 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:30.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:30.342 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:30.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:30.343 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.343 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:30.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:30.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:53:30.370 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:30.371 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:30.371 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:53:30.371 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55883 2020-03-11T22:53:30.373 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:30.373 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:53:30.374 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:30.375 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:30.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:30.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:30.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:30.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:30.377 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:30.377 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:30.377 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:30.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:30.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:30.378 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:30.378 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.379 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:30.380 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:30.381 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:30.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:53:30.381 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:53:30.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:30.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:30.382 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55883 2020-03-11T22:53:30.382 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55883 2020-03-11T22:53:30.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:30.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:30.384 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:53:30.385 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:53:30.385 INFO:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:53:30.420 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:53:30.420 INFO:teuthology.orchestra.run.smithi118:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:53:30.478 INFO:tasks.workunit.client.0.smithi118.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:53:30.489 INFO:tasks.workunit.client.0.smithi118.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:53:30.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:53:30.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:53:30.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:53:30.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:53:30.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:53:30.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:53:30.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:53:30.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:53:30.491 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:53:30.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:53:30.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:53:30.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:53:30.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:53:30.493 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:53:30.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none ' 2020-03-11T22:53:30.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:30.493 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:53:30.494 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:53:30.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:53:30.494 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_backfill_log_2 2020-03-11T22:53:30.494 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_recovery_1 2020-03-11T22:53:30.494 INFO:tasks.workunit.client.0.smithi118.stderr:TEST_recovery_2' 2020-03-11T22:53:30.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:53:30.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:53:30.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:30.495 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:30.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:30.496 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:30.496 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:30.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:30.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:30.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:30.497 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:30.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:30.504 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:30.504 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:30.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:30.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:30.505 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:30.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:30.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:30.506 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:30.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:30.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:30.507 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:30.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:30.507 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:30.508 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:30.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76637 2020-03-11T22:53:30.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:30.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:30.508 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:53:30.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:53:30.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:30.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:30.509 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76637 2020-03-11T22:53:30.509 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:53:30.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:53:30.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:53:30.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:53:30.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:53:30.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:53:30.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:53:30.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:30.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:53:30.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:53:30.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:53:30.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:53:30.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:53:30.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:53:30.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:53:30.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:30.814 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:53:31.014 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:53:31.016 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:53:31.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:31.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:31.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:53:31.017 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:53:31.018 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:31.018 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:31.018 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:31.018 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:31.019 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:31.019 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:31.019 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76637/ceph-mon.a.asok 2020-03-11T22:53:31.019 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:31.020 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76637/ceph-mon.a.asok config get fsid 2020-03-11T22:53:31.102 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:31.103 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76637/ceph-mon.a.asok 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:31.104 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76637/ceph-mon.a.asok config get mon_host 2020-03-11T22:53:31.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:53:31.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:31.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:53:31.188 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:53:31.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:53:31.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:53:31.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:53:32.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:32.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:32.066 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:32.066 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:32.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:53:32.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:32.086 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:32.087 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:53:32.087 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:53:32.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:32.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:53:32.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:32.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:32.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:53:32.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:32.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:53:32.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:32.095 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:32.096 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:32.097 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:32.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:32.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:53:32.098 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:32.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bbfca4b7-256b-4888-b25f-ccdfdd4e357b 2020-03-11T22:53:32.098 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 bbfca4b7-256b-4888-b25f-ccdfdd4e357b' 2020-03-11T22:53:32.099 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:32.099 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 bbfca4b7-256b-4888-b25f-ccdfdd4e357b 2020-03-11T22:53:32.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDsa2lemXQWBhAAfm9gl9hJidyUzy07WUh14w== 2020-03-11T22:53:32.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDsa2lemXQWBhAAfm9gl9hJidyUzy07WUh14w=="}' 2020-03-11T22:53:32.105 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bbfca4b7-256b-4888-b25f-ccdfdd4e357b -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:32.446 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:53:32.455 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:32.456 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDsa2lemXQWBhAAfm9gl9hJidyUzy07WUh14w== --osd-uuid bbfca4b7-256b-4888-b25f-ccdfdd4e357b 2020-03-11T22:53:32.480 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.477+0000 7f5a7934aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:32.481 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.479+0000 7f5a7934aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:32.483 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.480+0000 7f5a7934aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:32.483 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.481+0000 7f5a7934aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:32.484 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.481+0000 7f5a7934aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:32.484 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.481+0000 7f5a7934aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:32.522 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:32.519+0000 7f5a7934aec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:53:35.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:53:35.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:35.598 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:53:35.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:53:35.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:36.050 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:53:36.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:53:36.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:36.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:36.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:36.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:36.072 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:36.070+0000 7fa9cdf6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.074 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:36.071+0000 7fa9cdf6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.075 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:36.072+0000 7fa9cdf6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:53:36.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:36.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:53:36.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:36.395 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:53:36.595 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:36.592+0000 7fa9cdf6bec0 -1 Falling back to public interface 2020-03-11T22:53:36.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:37.389 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:37.386+0000 7fa9cdf6bec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:53:37.742 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:37.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:37.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:37.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:37.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:37.743 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:53:38.092 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:39.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:39.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:39.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:53:39.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:39.094 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:39.095 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:53:39.459 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77069,v1:127.0.0.1:6803/77069] [v2:127.0.0.1:6804/77069,v1:127.0.0.1:6805/77069] exists,up bbfca4b7-256b-4888-b25f-ccdfdd4e357b 2020-03-11T22:53:39.459 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 26703f65-af82-403c-ac35-78f94a9f76dd 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:53:39.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:39.461 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:39.462 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:39.463 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:39.464 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=26703f65-af82-403c-ac35-78f94a9f76dd 2020-03-11T22:53:39.465 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 26703f65-af82-403c-ac35-78f94a9f76dd' 2020-03-11T22:53:39.466 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:39.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDza2leMUILHBAAh/zwnDAo4yTF6ogRko8OJQ== 2020-03-11T22:53:39.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDza2leMUILHBAAh/zwnDAo4yTF6ogRko8OJQ=="}' 2020-03-11T22:53:39.473 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 26703f65-af82-403c-ac35-78f94a9f76dd -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:39.910 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:53:39.920 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:39.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDza2leMUILHBAAh/zwnDAo4yTF6ogRko8OJQ== --osd-uuid 26703f65-af82-403c-ac35-78f94a9f76dd 2020-03-11T22:53:39.944 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.941+0000 7f8894647ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:39.945 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.942+0000 7f8894647ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:39.947 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.944+0000 7f8894647ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:39.948 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.945+0000 7f8894647ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:39.948 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.945+0000 7f8894647ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:39.948 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:39.945+0000 7f8894647ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:40.005 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:40.002+0000 7f8894647ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:53:43.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:53:43.045 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:43.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:53:43.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:43.046 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:53:43.500 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:53:43.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:53:43.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:43.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:43.501 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:43.502 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:43.523 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:43.520+0000 7f1fe1d82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.524 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:43.522+0000 7f1fe1d82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.526 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:43.523+0000 7f1fe1d82ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:43.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:43.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:43.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:43.870 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:53:44.041 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:44.038+0000 7f1fe1d82ec0 -1 Falling back to public interface 2020-03-11T22:53:44.236 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:44.780 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:44.777+0000 7f1fe1d82ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:53:45.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:45.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:45.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:45.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:45.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:45.238 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:53:45.613 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77460,v1:127.0.0.1:6811/77460] [v2:127.0.0.1:6812/77460,v1:127.0.0.1:6813/77460] exists,up 26703f65-af82-403c-ac35-78f94a9f76dd 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:53:45.614 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:45.615 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:45.616 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:45.617 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:45.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:45.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:45.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:45.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:45.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:53:45.620 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:45.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3dd78f6c-924f-4a4f-8203-c71abbc456cc 2020-03-11T22:53:45.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 3dd78f6c-924f-4a4f-8203-c71abbc456cc' 2020-03-11T22:53:45.620 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:45.621 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 3dd78f6c-924f-4a4f-8203-c71abbc456cc 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD5a2le9CdNJRAAZU5+z/qu3VVCoZm3LrkxNQ== 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD5a2le9CdNJRAAZU5+z/qu3VVCoZm3LrkxNQ=="}' 2020-03-11T22:53:45.629 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3dd78f6c-924f-4a4f-8203-c71abbc456cc -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:46.135 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:53:46.145 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:46.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD5a2le9CdNJRAAZU5+z/qu3VVCoZm3LrkxNQ== --osd-uuid 3dd78f6c-924f-4a4f-8203-c71abbc456cc 2020-03-11T22:53:46.169 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.166+0000 7f4adff89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:46.170 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.168+0000 7f4adff89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:46.172 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.169+0000 7f4adff89ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:46.172 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.169+0000 7f4adff89ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:46.172 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.169+0000 7f4adff89ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:46.172 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.169+0000 7f4adff89ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:46.288 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:46.285+0000 7f4adff89ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:53:49.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:53:49.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:49.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:53:49.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:49.394 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:53:49.847 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:53:49.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:53:49.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:49.847 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:49.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:49.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:49.868 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:49.865+0000 7ff323e5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:49.870 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:49.867+0000 7ff323e5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:49.871 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:49.869+0000 7ff323e5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:50.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:53:50.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:50.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:53:50.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:50.190 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:53:50.399 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:50.396+0000 7ff323e5aec0 -1 Falling back to public interface 2020-03-11T22:53:50.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:51.230 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:51.227+0000 7ff323e5aec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:53:51.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:51.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:51.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:51.536 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:51.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:51.537 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:53:51.919 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:52.921 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:52.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:52.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:53:52.922 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:52.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:52.923 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:53:53.272 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77815,v1:127.0.0.1:6819/77815] [v2:127.0.0.1:6820/77815,v1:127.0.0.1:6821/77815] exists,up 3dd78f6c-924f-4a4f-8203-c71abbc456cc 2020-03-11T22:53:53.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:53.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:53.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:53.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:53:53.273 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:53.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:53.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:53.275 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:53.275 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:53.275 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:53.276 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:53.276 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:53.276 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:53.276 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:53.276 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:53.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:53.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:53.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:53.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:53.277 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:53.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:53.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:53.278 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:53.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:53.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:53.279 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:53:53.280 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:53.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d7135130-02af-4198-9673-c51d2c48fb99 2020-03-11T22:53:53.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 d7135130-02af-4198-9673-c51d2c48fb99' 2020-03-11T22:53:53.280 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:53.281 INFO:tasks.workunit.client.0.smithi118.stdout:add osd3 d7135130-02af-4198-9673-c51d2c48fb99 2020-03-11T22:53:53.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQABbGleM4PzEBAAZF3PcVkWIo4JjgPwqQfOUw== 2020-03-11T22:53:53.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQABbGleM4PzEBAAZF3PcVkWIo4JjgPwqQfOUw=="}' 2020-03-11T22:53:53.288 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d7135130-02af-4198-9673-c51d2c48fb99 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:53.737 INFO:tasks.workunit.client.0.smithi118.stdout:3 2020-03-11T22:53:53.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:53.748 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQABbGleM4PzEBAAZF3PcVkWIo4JjgPwqQfOUw== --osd-uuid d7135130-02af-4198-9673-c51d2c48fb99 2020-03-11T22:53:53.771 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.769+0000 7fe109e0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.797 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.794+0000 7fe109e0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.799 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.796+0000 7fe109e0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.800 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.797+0000 7fe109e0eec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.800 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.797+0000 7fe109e0eec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.800 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.797+0000 7fe109e0eec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.922 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:53.919+0000 7fe109e0eec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:53:57.029 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:53:57.029 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:57.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:53:57.030 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:57.030 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd3 key to auth repository 2020-03-11T22:53:57.493 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.3 2020-03-11T22:53:57.493 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:53:57.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:57.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:57.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:57.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:57.515 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:57.512+0000 7fe5bc8dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.517 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:57.514+0000 7fe5bc8dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.518 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:57.515+0000 7fe5bc8dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.860 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:53:57.860 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:57.860 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:53:57.860 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:57.861 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:53:58.032 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:58.029+0000 7fe5bc8dcec0 -1 Falling back to public interface 2020-03-11T22:53:58.209 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:58.748 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:53:58.744+0000 7fe5bc8dcec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:53:59.595 INFO:tasks.workunit.client.0.smithi118.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78198,v1:127.0.0.1:6827/78198] [v2:127.0.0.1:6828/78198,v1:127.0.0.1:6829/78198] exists,up d7135130-02af-4198-9673-c51d2c48fb99 2020-03-11T22:53:59.595 INFO:tasks.workunit.client.0.smithi118.stdout:add osd4 f291f4cb-1e51-4280-8119-2e657a433a56 2020-03-11T22:53:59.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:59.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:59.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:59.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:59.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:53:59.596 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:59.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:53:59.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:59.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:59.597 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:59.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:59.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:59.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:59.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:53:59.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:53:59.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:59.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:59.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:59.599 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:59.600 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:59.600 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:59.600 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:59.600 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.600 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:53:59.601 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:59.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:53:59.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:59.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:59.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:59.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:59.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:59.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:59.602 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:59.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:59.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:59.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:59.603 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:53:59.603 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:59.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f291f4cb-1e51-4280-8119-2e657a433a56 2020-03-11T22:53:59.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 f291f4cb-1e51-4280-8119-2e657a433a56' 2020-03-11T22:53:59.604 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:59.607 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAHbGlemqQMJBAA/4KqyfX2eNWyMca+fQy1BA== 2020-03-11T22:53:59.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAHbGlemqQMJBAA/4KqyfX2eNWyMca+fQy1BA=="}' 2020-03-11T22:53:59.608 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f291f4cb-1e51-4280-8119-2e657a433a56 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:00.171 INFO:tasks.workunit.client.0.smithi118.stdout:4 2020-03-11T22:54:00.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:00.182 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHbGlemqQMJBAA/4KqyfX2eNWyMca+fQy1BA== --osd-uuid f291f4cb-1e51-4280-8119-2e657a433a56 2020-03-11T22:54:00.206 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.202+0000 7f17f77e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:00.208 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.204+0000 7f17f77e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:00.209 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.205+0000 7f17f77e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:00.209 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.205+0000 7f17f77e0ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:00.209 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.205+0000 7f17f77e0ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:00.210 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.205+0000 7f17f77e0ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:00.306 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:00.303+0000 7f17f77e0ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:54:03.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:54:03.452 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:03.453 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd4 key to auth repository 2020-03-11T22:54:03.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:54:03.453 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:03.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:54:03.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:03.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:03.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:03.904 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:03.904 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.4 2020-03-11T22:54:03.925 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:03.922+0000 7fa124c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:03.927 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:03.923+0000 7fa124c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:03.928 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:03.924+0000 7fa124c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:04.252 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:54:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:54:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:04.253 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:04.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:04.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:04.254 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:04.254 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:54:04.445 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:04.441+0000 7fa124c1cec0 -1 Falling back to public interface 2020-03-11T22:54:04.599 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:05.208 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:05.204+0000 7fa124c1cec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:54:05.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:05.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:05.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:05.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:05.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:05.602 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi118.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78553,v1:127.0.0.1:6835/78553] [v2:127.0.0.1:6836/78553,v1:127.0.0.1:6837/78553] exists,up f291f4cb-1e51-4280-8119-2e657a433a56 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e61e13f2-9a82-45af-9c48-c8bb40f48109 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 e61e13f2-9a82-45af-9c48-c8bb40f48109' 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi118.stdout:add osd5 e61e13f2-9a82-45af-9c48-c8bb40f48109 2020-03-11T22:54:06.002 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQANbGle4aiMOxAABl6s2BHk1gKFoCMyIWNpCA== 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQANbGle4aiMOxAABl6s2BHk1gKFoCMyIWNpCA=="}' 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e61e13f2-9a82-45af-9c48-c8bb40f48109 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:06.531 INFO:tasks.workunit.client.0.smithi118.stdout:5 2020-03-11T22:54:06.541 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:06.542 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQANbGle4aiMOxAABl6s2BHk1gKFoCMyIWNpCA== --osd-uuid e61e13f2-9a82-45af-9c48-c8bb40f48109 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.562+0000 7f75c5ca9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.568 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.564+0000 7f75c5ca9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.568 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.565+0000 7f75c5ca9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.569 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.565+0000 7f75c5ca9ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.569 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.565+0000 7f75c5ca9ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.569 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.565+0000 7f75c5ca9ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.667 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:06.664+0000 7f75c5ca9ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:54:10.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:54:10.007 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:10.008 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd5 key to auth repository 2020-03-11T22:54:10.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:54:10.008 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:10.459 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.5 2020-03-11T22:54:10.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:54:10.459 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:10.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:10.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:10.460 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:10.481 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:10.477+0000 7ff8ed58aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:10.482 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:10.479+0000 7ff8ed58aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:10.484 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:10.480+0000 7ff8ed58aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:10.805 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:10.806 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:10.807 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:10.807 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:54:11.001 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:10.996+0000 7ff8ed58aec0 -1 Falling back to public interface 2020-03-11T22:54:11.146 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:11.785 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:54:11.781+0000 7ff8ed58aec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:54:12.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:12.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:12.148 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:12.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:12.149 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:12.149 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:54:12.503 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:13.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:13.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:13.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:54:13.505 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:13.506 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:13.506 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:54:13.854 INFO:tasks.workunit.client.0.smithi118.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/78908,v1:127.0.0.1:6843/78908] [v2:127.0.0.1:6844/78908,v1:127.0.0.1:6845/78908] exists,up e61e13f2-9a82-45af-9c48-c8bb40f48109 2020-03-11T22:54:13.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:13.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:13.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:13.854 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:54:13.855 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:54:14.478 INFO:tasks.workunit.client.0.smithi118.stderr:pool 'test' already exists 2020-03-11T22:54:14.486 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:54:15.488 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:54:15.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.489 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:54:15.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.537 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:54:15.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:15.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:15.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.835 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:15.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.969 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:16.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:16.053 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:16.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.095 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:16.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.162 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:16.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:16.353 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.353 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:16.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.478 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:16.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.604 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:16.645 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:16.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.689 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:16.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.729 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:16.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.771 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:16.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.938 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:17.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.022 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:17.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.062 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:17.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:17.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.204 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:17.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.281 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:17.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:17.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.598 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:17.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.646 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:17.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:17.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:17.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:17.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.856 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:18.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.033 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:18.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.126 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:18.171 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.171 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:18.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:18.271 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.272 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:18.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.358 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:18.405 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.405 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:18.525 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.525 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:18.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.651 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:54:18.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.697 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:54:18.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.738 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:54:18.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:54:18.821 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.822 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:54:18.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.863 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:54:18.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.993 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:54:19.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.038 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:54:19.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.135 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:54:19.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:54:19.261 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.262 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:54:19.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.322 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:54:19.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:54:19.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.611 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:54:19.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.655 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:54:19.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.753 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:54:19.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.848 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:54:19.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.929 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:54:19.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.980 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.104 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:54:20.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.155 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:54:20.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.287 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:54:20.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:54:20.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:54:20.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:54:20.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.490 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:54:20.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.638 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:54:20.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:54:20.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.780 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:54:20.832 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.832 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:54:20.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.961 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:54:21.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.024 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:54:21.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:54:21.180 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.181 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:54:21.233 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.233 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:54:21.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.390 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:54:21.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.441 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:54:21.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:54:21.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:54:21.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.639 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:54:21.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.749 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:54:21.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:54:22.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.006 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:54:22.057 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.058 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:54:22.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.108 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:54:22.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.158 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:54:22.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:54:22.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.399 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:54:22.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.534 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:54:22.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:54:22.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:54:22.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.741 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:54:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:54:22.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.902 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:54:22.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.972 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:54:23.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.069 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:54:23.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.125 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:54:23.175 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.175 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:54:23.224 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.225 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:54:23.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.308 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:54:23.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.428 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:54:23.529 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.530 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:54:23.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.583 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:54:23.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.649 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:54:23.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.700 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:54:23.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.750 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:54:23.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:54:23.957 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.958 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:54:24.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:54:24.116 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:54:24.166 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.167 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:54:24.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:54:24.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.267 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:54:24.333 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.334 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:54:24.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:54:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.539 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:54:24.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:54:24.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.667 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:54:24.717 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.717 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:54:24.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.768 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:54:24.817 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.817 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:54:24.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:24.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:54:25.031 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.032 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:54:25.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.084 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:54:25.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.134 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:54:25.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.184 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:54:25.234 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.234 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:54:25.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.317 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:54:25.440 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.441 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:54:25.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:54:25.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.610 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:54:25.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:54:25.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.726 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:54:25.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.776 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:54:25.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:54:25.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:25.942 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:54:26.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.042 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:54:26.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.093 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:54:26.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.143 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:54:26.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.193 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:54:26.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.243 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:54:26.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:54:26.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:54:26.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:54:26.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.568 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:54:26.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.618 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:54:26.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:54:26.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:26.743 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:54:26.794 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:26.794 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:27.111 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:54:27.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:27.939 INFO:tasks.workunit.client.0.smithi118.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:27.947 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:27.948 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:28.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:28.949 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:28.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:28.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:28.950 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:28.951 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:29.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:29.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:29.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:29.051 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:29.052 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:1 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:2 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:3 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:4 2020-03-11T22:54:29.425 INFO:tasks.workunit.client.0.smithi118.stderr:5' 2020-03-11T22:54:29.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:29.426 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:29.426 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:29.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:54:29.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:54:29.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:29.550 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:29.673 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:54:29.673 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2020-03-11T22:54:29.673 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:29.674 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:29.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:54:29.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450' 2020-03-11T22:54:29.798 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:29.799 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:29.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345929 2020-03-11T22:54:29.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929' 2020-03-11T22:54:29.923 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:29.924 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:30.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:54:30.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182408' 2020-03-11T22:54:30.046 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.047 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:30.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051590 2020-03-11T22:54:30.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182408 5-124554051590' 2020-03-11T22:54:30.170 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:30.171 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:30.171 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:30.172 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:30.173 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:30.173 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:30.174 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:54:30.174 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:54:30.174 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:54:30.175 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:30.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:54:30.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:31.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:31.518 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:31.866 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:54:31.867 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:31.867 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:54:31.868 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:31.868 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:31.869 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:54:31.869 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:31.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:54:31.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:54:31.871 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:54:31.871 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:32.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672971 2020-03-11T22:54:32.213 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:32.214 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:54:32.214 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:32.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:32.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:54:32.216 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:32.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:54:32.217 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:54:32.217 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:54:32.217 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:32.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509450 2020-03-11T22:54:32.561 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:32.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345929 2020-03-11T22:54:32.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:32.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:32.564 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345929 2020-03-11T22:54:32.564 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:32.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345929 2020-03-11T22:54:32.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2020-03-11T22:54:32.565 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.3 seq 85899345929 2020-03-11T22:54:32.566 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:33.116 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345929 2020-03-11T22:54:33.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:33.117 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:54:33.117 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:33.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:33.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:54:33.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:33.120 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:54:33.120 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:54:33.121 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:54:33.121 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:33.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182409 -lt 107374182408 2020-03-11T22:54:33.492 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:33.493 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-124554051590 2020-03-11T22:54:33.493 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:33.494 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:54:33.495 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-124554051590 2020-03-11T22:54:33.495 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:33.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051590 2020-03-11T22:54:33.496 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 124554051590' 2020-03-11T22:54:33.496 INFO:tasks.workunit.client.0.smithi118.stdout:waiting osd.5 seq 124554051590 2020-03-11T22:54:33.497 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:54:33.842 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051591 -lt 124554051590 2020-03-11T22:54:33.843 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:33.843 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:33.843 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:34.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:54:34.286 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:34.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:34.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:34.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:34.287 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:34.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:34.288 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:34.676 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:35.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:35.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:54:35.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:35.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:54:35.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:35.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:35.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:35.223 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:35.224 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:35.224 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:35.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:35.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:35.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:35.225 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:35.558 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:35.558 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:35.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:35.559 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:36.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:36.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:36.003 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:36.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:36.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:36.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:36.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:36.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:36.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:36.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23348 2020-03-11T22:54:36.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23348 '!=' null 2020-03-11T22:54:36.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:36.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:36.471 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:36.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:36.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:36.572 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:36.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:36.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:36.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:36.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:36.573 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:36.903 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:36.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:36.904 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:36.905 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:37.350 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:37.351 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:37.352 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:37.352 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:37.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18724 2020-03-11T22:54:37.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18724 '!=' null 2020-03-11T22:54:37.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:37.799 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:37.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:37.899 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:37.900 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:37.901 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:38.237 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:38.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:38.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:38.238 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:38.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:38.686 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:38.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:38.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:38.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:38.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:38.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:38.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:38.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:39.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18724 2020-03-11T22:54:39.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18724 '!=' null 2020-03-11T22:54:39.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:39.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:39.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:39.289 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:39.290 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:39.291 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:39.624 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:39.625 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:39.625 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:39.626 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:40.068 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:40.069 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:40.069 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:40.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34147 2020-03-11T22:54:40.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34147 '!=' null 2020-03-11T22:54:40.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:40.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:40.520 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:40.619 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:40.620 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:40.621 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:40.952 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:40.952 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:40.953 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:40.953 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:41.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:41.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:41.400 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:41.401 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:41.401 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:41.401 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:41.401 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:41.401 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:41.402 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:41.869 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:54:41.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:54:41.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:41.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:41.870 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:41.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:41.970 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:41.972 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:41.972 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:41.972 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:41.972 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:41.973 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:41.973 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:42.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:42.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:42.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:42.303 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:42.747 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:42.748 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:42.748 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:42.748 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:43.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:54:43.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:54:43.231 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:43.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:43.232 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:43.331 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:43.332 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:43.333 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:43.687 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:43.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:43.688 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:43.689 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:44.129 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:44.130 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:44.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:44.131 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:44.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:54:44.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:54:44.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:44.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:44.578 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:44.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:44.678 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:44.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:44.679 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:44.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:44.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:44.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:44.680 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:45.010 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:45.011 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:45.011 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:45.012 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:45.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:45.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:45.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:45.457 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:45.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:45.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:45.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:45.458 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:45.459 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:45.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=48873 2020-03-11T22:54:45.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 48873 '!=' null 2020-03-11T22:54:45.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:45.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:45.912 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:46.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:46.012 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:46.013 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:46.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:46.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:46.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:46.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:46.014 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:46.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:46.344 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:46.345 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:46.345 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:46.791 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:46.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:46.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:46.808 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:46.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:46.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:46.809 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:46.809 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:46.810 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:47.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=48873 2020-03-11T22:54:47.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 48873 '!=' null 2020-03-11T22:54:47.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:47.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:47.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:47.338 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:47.339 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:47.340 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:47.674 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:47.675 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:47.675 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:47.675 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:48.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:48.122 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:48.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:48.124 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:48.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34498 2020-03-11T22:54:48.571 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34498 '!=' null 2020-03-11T22:54:48.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:48.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:48.572 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:48.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:48.672 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:48.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:48.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:48.673 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:48.674 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:48.674 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:48.674 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:49.004 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:49.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:49.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:49.005 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:49.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:49.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:49.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:49.483 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:49.483 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:49.934 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=52323 2020-03-11T22:54:49.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 52323 '!=' null 2020-03-11T22:54:49.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:49.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:49.935 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:50.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:50.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:50.037 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:50.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:50.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:50.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:50.377 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:50.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:50.826 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:50.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:50.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:50.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:50.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:50.827 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:50.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:50.828 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:51.274 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=52323 2020-03-11T22:54:51.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 52323 '!=' null 2020-03-11T22:54:51.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:51.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:51.275 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:51.374 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:51.375 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:51.376 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:51.709 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:51.710 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:51.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:51.711 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:52.156 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:52.157 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:52.158 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:52.158 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:52.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=35648 2020-03-11T22:54:52.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 35648 '!=' null 2020-03-11T22:54:52.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:52.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:52.601 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:52.701 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:52.702 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:52.702 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:52.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:52.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:52.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:52.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:52.703 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:53.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:53.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:53.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:53.036 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:53.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:53.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:53.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:53.482 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:53.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:53.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:53.483 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:53.483 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:53.484 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:53.963 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=35648 2020-03-11T22:54:53.964 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 35648 '!=' null 2020-03-11T22:54:53.964 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:53.964 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:53.964 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:54.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:54.063 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:54.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:54.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:54.064 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:54.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:54.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:54.065 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:54.395 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:54.395 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:54.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:54.396 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:54.846 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:54.846 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:55.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=35648 2020-03-11T22:54:55.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 35648 '!=' null 2020-03-11T22:54:55.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:55.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:55.330 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:55.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:55.430 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:55.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:55.431 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:55.432 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:55.800 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:55.801 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:55.801 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:55.801 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:56.238 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:56.239 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:56.240 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:56.240 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:56.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=52610 2020-03-11T22:54:56.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 52610 '!=' null 2020-03-11T22:54:56.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:56.711 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:56.712 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:56.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:56.812 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:56.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:56.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:56.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:56.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:56.813 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:56.814 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:57.139 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:57.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:57.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:57.140 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:57.592 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:57.593 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:57.593 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:57.593 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:58.082 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29323 2020-03-11T22:54:58.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29323 '!=' null 2020-03-11T22:54:58.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:58.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:58.083 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:58.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:58.183 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:58.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:58.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:58.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:58.184 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:58.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:58.185 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:58.515 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:58.515 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:58.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:58.516 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:58.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:54:58.954 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:58.955 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:54:59.272 INFO:tasks.workunit.client.0.smithi118.stderr:dumped pgs 2020-03-11T22:54:59.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:54:59.280 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:54:59.281 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:59.282 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:59.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:59.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:59.282 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:59.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:59.392 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:54:59.393 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:54:59.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:59.394 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout:{ 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_log_t": { 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: "head": "33'150", 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: "tail": "33'148", 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: "log": [ 2020-03-11T22:55:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "modify", 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'149", 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "prior_version": "0'0", 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4637.0:1", 2020-03-11T22:55:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: "extra_reqids": [], 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "mtime": "2020-03-11T22:54:26.712404+0000", 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": 0, 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "mod_desc": { 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "object_mod_desc": { 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "can_local_rollback": false, 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "rollback_info_completed": false, 2020-03-11T22:55:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: "ops": [] 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_regions": { 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: "object_clean_regions": { 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_omap": true, 2020-03-11T22:55:00.481 INFO:tasks.workunit.client.0.smithi118.stdout: "new_object": false 2020-03-11T22:55:00.483 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.483 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.483 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.483 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.483 INFO:tasks.workunit.client.0.smithi118.stdout: "op": "modify", 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'150", 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "prior_version": "0'0", 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4640.0:1", 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "extra_reqids": [], 2020-03-11T22:55:00.484 INFO:tasks.workunit.client.0.smithi118.stdout: "mtime": "2020-03-11T22:54:26.762220+0000", 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": 0, 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "mod_desc": { 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "object_mod_desc": { 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "can_local_rollback": false, 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "rollback_info_completed": false, 2020-03-11T22:55:00.485 INFO:tasks.workunit.client.0.smithi118.stdout: "ops": [] 2020-03-11T22:55:00.486 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.486 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.486 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_regions": { 2020-03-11T22:55:00.486 INFO:tasks.workunit.client.0.smithi118.stdout: "object_clean_regions": { 2020-03-11T22:55:00.486 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: "clean_omap": true, 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: "new_object": false 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: ], 2020-03-11T22:55:00.487 INFO:tasks.workunit.client.0.smithi118.stdout: "dups": [ 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'141", 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "141", 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.488 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'142", 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "142", 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.489 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'143", 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "143", 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.490 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'144", 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "144", 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.491 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.492 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:55:00.492 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'145", 2020-03-11T22:55:00.492 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "145", 2020-03-11T22:55:00.492 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.493 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.493 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.493 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'146", 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "146", 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.494 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'147", 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "147", 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: { 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:55:00.495 INFO:tasks.workunit.client.0.smithi118.stdout: "version": "33'148", 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: "user_version": "148", 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: "return_code": "0" 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: ] 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: }, 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: "pg_missing_t": { 2020-03-11T22:55:00.496 INFO:tasks.workunit.client.0.smithi118.stdout: "missing": [], 2020-03-11T22:55:00.497 INFO:tasks.workunit.client.0.smithi118.stdout: "may_include_deletes": true 2020-03-11T22:55:00.497 INFO:tasks.workunit.client.0.smithi118.stdout: } 2020-03-11T22:55:00.497 INFO:tasks.workunit.client.0.smithi118.stdout:} 2020-03-11T22:55:00.498 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:00.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:55:00.498 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to max 2 2020-03-11T22:55:00.499 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to copy log version 33'150 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to copy log version 33'149 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:55:00.500 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to max 2 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to copy log version 33'150 2020-03-11T22:55:00.501 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to copy log version 33'149 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:55:00.502 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:55:00.503 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:55:00.503 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.429+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:55:00.503 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to max 2 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to copy log version 33'150 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to copy log version 33'149 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:55:00.504 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:55:00.506 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:55:00.506 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:55:00.506 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:55:00.506 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:55:00.507 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:28.430+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:55:00.507 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_after v 33'150 2020-03-11T22:55:00.507 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:55:00.507 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:55:00.508 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:55:00.508 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:55:00.508 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'149 2020-03-11T22:55:00.509 INFO:tasks.workunit.client.0.smithi118.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:55.183+0000 7fa9ac126700 20 copy_up_to/copy_after copy dup from log version 33'150 2020-03-11T22:55:00.510 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:00.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:55:00.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:00.511 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:00.512 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:00.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:00.513 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:00.513 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:00.514 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:00.514 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:00.516 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:00.517 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:00.517 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:00.518 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:00.518 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:00.519 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:00.519 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:00.521 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:00.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:00.522 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:00.522 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:00.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:00.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:00.523 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:00.547 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:00.547 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:00.547 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:00.547 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76637 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:00.548 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:00.549 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:00.550 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:00.550 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:00.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:00.551 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:00.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:00.551 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:00.551 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:00.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:00.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:00.552 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:00.552 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:00.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:00.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:00.553 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:00.554 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:00.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:00.555 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:00.559 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76637 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:00.562 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76637 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:55:00.563 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:55:00.564 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:55:00.565 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:55:00.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:55:00.566 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.898 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:55:00.899 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:00.899 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:00.899 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:00.899 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:00.899 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:00.900 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:00.900 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:55:01.107 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:55:01.109 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:01.110 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:01.111 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:01.111 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.111 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:01.111 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76637/ceph-mon.a.asok 2020-03-11T22:55:01.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:01.112 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76637/ceph-mon.a.asok config get fsid 2020-03-11T22:55:01.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:55:01.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:01.205 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:01.206 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.207 INFO:tasks.workunit.client.0.smithi118.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:01.207 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76637/ceph-mon.a.asok 2020-03-11T22:55:01.207 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:01.207 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76637/ceph-mon.a.asok config get mon_host 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:55:02.166 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:02.167 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:02.168 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:02.168 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:02.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:55:02.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:02.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:02.191 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:55:02.191 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:55:02.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:02.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:55:02.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:02.196 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:02.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:55:02.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:02.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:55:02.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:02.197 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:02.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:02.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:02.198 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:02.199 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:02.200 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:02.201 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:02.202 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6 2020-03-11T22:55:02.203 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6' 2020-03-11T22:55:02.203 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:02.203 INFO:tasks.workunit.client.0.smithi118.stdout:add osd0 f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6 2020-03-11T22:55:02.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBGbGlemHxGDBAAOPjfy9STQWNhBD9ASuqXiA== 2020-03-11T22:55:02.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBGbGlemHxGDBAAOPjfy9STQWNhBD9ASuqXiA=="}' 2020-03-11T22:55:02.215 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:02.546 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:55:02.556 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:02.557 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGbGlemHxGDBAAOPjfy9STQWNhBD9ASuqXiA== --osd-uuid f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6 2020-03-11T22:55:02.580 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.577+0000 7f5634fabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:02.582 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.579+0000 7f5634fabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:02.583 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.580+0000 7f5634fabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:02.584 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.581+0000 7f5634fabec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:02.584 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.581+0000 7f5634fabec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:02.584 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.581+0000 7f5634fabec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:02.615 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:02.612+0000 7f5634fabec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:55:05.626 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:55:05.627 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:05.627 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:55:05.628 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:05.628 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd0 key to auth repository 2020-03-11T22:55:06.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:55:06.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:06.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:06.073 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:06.074 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:06.074 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.0 2020-03-11T22:55:06.094 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:06.091+0000 7f3d5eccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:06.095 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:06.093+0000 7f3d5eccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:06.096 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:06.094+0000 7f3d5eccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:06.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:55:06.417 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:06.418 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:06.419 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:55:06.612 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:06.609+0000 7f3d5eccdec0 -1 Falling back to public interface 2020-03-11T22:55:06.761 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:07.400 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:07.397+0000 7f3d5eccdec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:55:07.762 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:07.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:07.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:07.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:07.763 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:07.763 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:55:08.111 INFO:tasks.workunit.client.0.smithi118.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85194,v1:127.0.0.1:6803/85194] [v2:127.0.0.1:6804/85194,v1:127.0.0.1:6805/85194] exists,up f1f5cf7a-7ea2-4b14-b75b-8b1f4cdb42f6 2020-03-11T22:55:08.111 INFO:tasks.workunit.client.0.smithi118.stdout:add osd1 4c177bc5-f5a6-40a3-a304-b426b21af062 2020-03-11T22:55:08.111 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:08.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:08.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:08.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:08.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:55:08.112 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:08.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:08.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:55:08.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:08.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:55:08.113 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:55:08.114 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:55:08.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:08.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:08.115 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:08.115 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:08.116 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:08.116 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:08.116 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:08.116 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:08.117 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:08.118 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:55:08.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:08.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4c177bc5-f5a6-40a3-a304-b426b21af062 2020-03-11T22:55:08.119 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 4c177bc5-f5a6-40a3-a304-b426b21af062' 2020-03-11T22:55:08.119 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBMbGle/cE3BxAAHwL/jYp70mclsQVi+iFuOA== 2020-03-11T22:55:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBMbGle/cE3BxAAHwL/jYp70mclsQVi+iFuOA=="}' 2020-03-11T22:55:08.124 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4c177bc5-f5a6-40a3-a304-b426b21af062 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:08.633 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:55:08.643 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:08.644 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMbGle/cE3BxAAHwL/jYp70mclsQVi+iFuOA== --osd-uuid 4c177bc5-f5a6-40a3-a304-b426b21af062 2020-03-11T22:55:08.667 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.664+0000 7f7fe0d8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.701 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.696+0000 7f7fe0d8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.698+0000 7f7fe0d8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.699+0000 7f7fe0d8eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:08.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.699+0000 7f7fe0d8eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:08.702 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.699+0000 7f7fe0d8eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:08.766 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:08.763+0000 7f7fe0d8eec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:55:11.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:55:11.892 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:11.893 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd1 key to auth repository 2020-03-11T22:55:11.893 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:55:11.894 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.1 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:12.344 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:12.365 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:12.362+0000 7f7640d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.367 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:12.364+0000 7f7640d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.368 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:12.365+0000 7f7640d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:55:12.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:12.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:12.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:12.694 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:55:12.886 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:12.882+0000 7f7640d50ec0 -1 Falling back to public interface 2020-03-11T22:55:13.033 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:13.733 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:13.730+0000 7f7640d50ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:55:14.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:14.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:14.034 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:14.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:14.035 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:14.035 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:55:14.444 INFO:tasks.workunit.client.0.smithi118.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85549,v1:127.0.0.1:6811/85549] [v2:127.0.0.1:6812/85549,v1:127.0.0.1:6813/85549] exists,up 4c177bc5-f5a6-40a3-a304-b426b21af062 2020-03-11T22:55:14.445 INFO:tasks.workunit.client.0.smithi118.stdout:add osd2 2a48a65a-5a4b-434b-986c-71cb9f004d92 2020-03-11T22:55:14.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:14.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:14.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:14.445 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:55:14.446 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:14.447 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:14.448 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:14.449 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:14.450 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2a48a65a-5a4b-434b-986c-71cb9f004d92 2020-03-11T22:55:14.451 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 2a48a65a-5a4b-434b-986c-71cb9f004d92' 2020-03-11T22:55:14.451 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:14.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBSbGlel1cgGxAAVlKxliV2MjvD3wAkjIBxAA== 2020-03-11T22:55:14.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBSbGlel1cgGxAAVlKxliV2MjvD3wAkjIBxAA=="}' 2020-03-11T22:55:14.458 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2a48a65a-5a4b-434b-986c-71cb9f004d92 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:14.888 INFO:tasks.workunit.client.0.smithi118.stdout:2 2020-03-11T22:55:14.898 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:14.899 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBSbGlel1cgGxAAVlKxliV2MjvD3wAkjIBxAA== --osd-uuid 2a48a65a-5a4b-434b-986c-71cb9f004d92 2020-03-11T22:55:14.922 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.919+0000 7fe51f8d2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.923 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.920+0000 7fe51f8d2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.925 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.922+0000 7fe51f8d2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.926 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.922+0000 7fe51f8d2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:14.926 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.922+0000 7fe51f8d2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:14.926 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:14.922+0000 7fe51f8d2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:15.034 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:15.031+0000 7fe51f8d2ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:55:18.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:55:18.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:18.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:55:18.144 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:18.145 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd2 key to auth repository 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stdout:start osd.2 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:18.595 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:18.613+0000 7f0f6f952ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:18.624 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:18.614+0000 7f0f6f952ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:18.625 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:18.616+0000 7f0f6f952ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:18.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:55:18.955 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:18.957 INFO:tasks.workunit.client.0.smithi118.stdout:0 2020-03-11T22:55:19.137 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:19.133+0000 7f0f6f952ec0 -1 Falling back to public interface 2020-03-11T22:55:19.297 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:19.890 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:19.888+0000 7f0f6f952ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:55:20.302 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:20.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:20.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:20.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:20.304 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:20.305 INFO:tasks.workunit.client.0.smithi118.stdout:1 2020-03-11T22:55:20.690 INFO:tasks.workunit.client.0.smithi118.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85904,v1:127.0.0.1:6819/85904] [v2:127.0.0.1:6820/85904,v1:127.0.0.1:6821/85904] exists,up 2a48a65a-5a4b-434b-986c-71cb9f004d92 2020-03-11T22:55:20.690 INFO:tasks.workunit.client.0.smithi118.stdout:add osd3 df27fe99-7687-424b-8c0e-7158217b2443 2020-03-11T22:55:20.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:20.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:20.690 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:20.691 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:55:20.692 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:20.693 INFO:tasks.workunit.client.0.smithi118.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76637 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:20.694 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:55:20.695 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:20.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=df27fe99-7687-424b-8c0e-7158217b2443 2020-03-11T22:55:20.696 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 df27fe99-7687-424b-8c0e-7158217b2443' 2020-03-11T22:55:20.696 INFO:tasks.workunit.client.0.smithi118.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:20.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBYbGleB3XFKRAAcT5711Z4pdYWbCU1m0ukpQ== 2020-03-11T22:55:20.703 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBYbGleB3XFKRAAcT5711Z4pdYWbCU1m0ukpQ=="}' 2020-03-11T22:55:20.704 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new df27fe99-7687-424b-8c0e-7158217b2443 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:21.179 INFO:tasks.workunit.client.0.smithi118.stdout:3 2020-03-11T22:55:21.189 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:21.190 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=959ae9dc-7e89-48a3-a94b-4e7602851d3b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76637/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBYbGleB3XFKRAAcT5711Z4pdYWbCU1m0ukpQ== --osd-uuid df27fe99-7687-424b-8c0e-7158217b2443 2020-03-11T22:55:21.213 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.210+0000 7f01b6f30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.224 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.221+0000 7f01b6f30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.226 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.223+0000 7f01b6f30ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.226 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.223+0000 7f01b6f30ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:21.227 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.223+0000 7f01b6f30ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:21.227 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.223+0000 7f01b6f30ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:21.345 INFO:tasks.workunit.client.0.smithi118.stderr:2020-03-11T22:55:21.342+0000 7f01b6f30ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:55:24.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:55:24.584 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:24.585 INFO:tasks.workunit.client.0.smithi118.stdout:adding osd3 key to auth repository 2020-03-11T22:55:24.585 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:55:24.586 INFO:tasks.workunit.client.0.smithi118.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:25.001 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:25.001 DEBUG:teuthology.task.console_log:Killing console logger for smithi118 2020-03-11T22:55:25.001 DEBUG:teuthology.task.console_log:Killing console logger for smithi118 2020-03-11T22:55:25.002 DEBUG:teuthology.exit:Finished running handlers